Buckets:
| (MainProcess pid=132001) INFO 10-31 23:23:34 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'gpu_memory_utilization': 0.85, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'Qwen/Qwen3.5-4B'} | |
| (MainProcess pid=132001) INFO 10-31 23:23:35 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration | |
| (MainProcess pid=132001) INFO 10-31 23:23:35 [model.py:2152] Using max model len 32768 | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1116.84it/s] | |
| (MainProcess pid=132001) INFO 10-31 23:23:37 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=8192. | |
| (MainProcess pid=132001) INFO 10-31 23:23:37 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 1.59it/s] Parse safetensors files: 100%|██████████| 2/2 [00:01<00:00, 1.93it/s] Parse safetensors files: 100%|██████████| 2/2 [00:01<00:00, 1.87it/s] | |
| (MainProcess pid=132001) INFO 10-31 23:23:39 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=132001) WARNING 10-31 23:23:51 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized | |
| (EngineCore pid=132425) INFO 10-31 23:24:01 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=132425) INFO 10-31 23:24:03 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_b297644720bd49ff9e6e2c57bb9fa09e backend=nccl | |
| (EngineCore pid=132425) INFO 10-31 23:24:03 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=132425) INFO 10-31 23:24:03 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=132425) INFO 10-31 23:24:04 [model_runner.py:392] Loading model from scratch... | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1182.02it/s] | |
| (EngineCore pid=132425) INFO 10-31 23:24:05 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention | |
| (EngineCore pid=132425) INFO 10-31 23:24:05 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. | |
| (EngineCore pid=132425) INFO 10-31 23:24:05 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=132425) INFO 10-31 23:24:06 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=132425) INFO 10-31 23:24:06 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=132425) INFO 10-31 23:24:06 [base_loader.py:82] Initializing model took 2.143799 seconds | |
| (EngineCore pid=132425) INFO 10-31 23:24:07 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 133.92 GiB. | |
| (EngineCore pid=132425) INFO 10-31 23:24:07 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.53it/s] | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=132425) WARNING 10-31 23:24:07 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.61it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.59it/s] | |
| (EngineCore pid=132425) INFO 10-31 23:24:07 [default_loader.py:501] Loading weights took 0.78 seconds | |
| (EngineCore pid=132425) INFO 10-31 23:24:09 [model_runner.py:424] Model loading took 8.56 GiB memory and 4.867535 seconds | |
| (EngineCore pid=132425) INFO 10-31 23:24:09 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=132425) INFO 10-31 23:24:09 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=132425) INFO 10-31 23:24:09 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=132425) INFO 10-31 23:24:09 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=132425) INFO 10-31 23:24:09 [utils.py:347] Using LBNHC KV cache layout. | |
| [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning. | |
| (MainProcess pid=132001) INFO 10-31 23:24:12 [base.py:280] Multi-modal warmup completed in 21.180s | |
| (EngineCore pid=132425) INFO 10-31 23:24:17 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. | |
| (EngineCore pid=132425) INFO 10-31 23:24:21 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=13 num_submods=33 | |
| (EngineCore pid=132425) INFO 10-31 23:24:21 [decorators.py:319] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/bc5c0f9b4b3ba2fb87d47f0555b03ce07cf188ddf58e863bbd7a50dec0d4af47/rank_0_0/model | |
| (EngineCore pid=132425) INFO 10-31 23:24:21 [monitor.py:62] [backbone] torch.compile took 1.04 s in total | |
| (EngineCore pid=132425) INFO 10-31 23:24:24 [monitor.py:91] [backbone] Initial profiling/warmup run took 2.74 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:27, 1.82it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:21, 2.24it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:01<00:19, 2.50it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:01<00:19, 2.40it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:02<00:23, 2.00it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:02<00:24, 1.86it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:03<00:23, 1.84it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:04<00:22, 1.90it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:20, 2.00it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:04<00:19, 2.08it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:05<00:16, 2.38it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:05<00:14, 2.62it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:05<00:13, 2.79it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:06<00:12, 2.99it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:06<00:11, 3.23it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:06<00:09, 3.54it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:06<00:06, 5.41it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:06<00:04, 7.27it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:06<00:03, 9.09it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:07<00:02, 10.59it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:07<00:02, 11.86it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:07<00:01, 13.01it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:07<00:01, 13.74it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:07<00:01, 14.34it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:07<00:01, 14.25it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:07<00:01, 14.81it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:07<00:00, 15.24it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:08<00:00, 15.96it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:08<00:00, 16.68it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:08<00:00, 17.34it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:08<00:00, 17.75it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:08<00:00, 18.07it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:08<00:00, 18.30it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:08<00:00, 5.90it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 24.85it/s] | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [model_runner.py:1127] Graph capturing finished in 10 secs, took 0.40 GiB | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [gpu_worker.py:759] Available KV cache memory: 56.26 GiB | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.8500 is equivalent to --gpu-memory-utilization=0.8440 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.8560. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [kv_cache_utils.py:2327] GPU KV cache size: 1,657,870 tokens, Maximum concurrency for 32,768 tokens per request: 50.59x | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)]. | |
| (EngineCore pid=132425) INFO 10-31 23:24:35 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:35, 1.41it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:38, 1.29it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:37, 1.28it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:35, 1.33it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:33, 1.39it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:30, 1.47it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:28, 1.56it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:25, 1.69it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:05<00:22, 1.84it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:20, 2.00it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:17, 2.24it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:15, 2.50it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:14, 2.66it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:12, 2.89it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:11, 3.15it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:07<00:09, 3.64it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:07<00:05, 5.61it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:08<00:04, 7.47it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:08<00:03, 9.18it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:08<00:02, 10.59it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:08<00:02, 11.61it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:01, 12.49it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:08<00:01, 13.02it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:08<00:01, 13.45it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:09<00:01, 13.62it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:09<00:01, 13.95it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:09<00:00, 14.25it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:09<00:00, 14.38it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:09<00:00, 14.59it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:09<00:00, 14.56it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:09<00:00, 14.59it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:09<00:00, 14.69it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:10<00:00, 14.71it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 5.00it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 21.16it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 21.05it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 21.15it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:01, 21.05it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 21.16it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 21.15it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 21.21it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 21.08it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 21.30it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 21.45it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 21.55it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 21.33it/s] | |
| (EngineCore pid=132425) INFO 10-31 23:24:48 [model_runner.py:1127] Graph capturing finished in 13 secs, took 0.24 GiB | |
| (EngineCore pid=132425) INFO 10-31 23:24:48 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (100.8%). | |
| (EngineCore pid=132425) INFO 10-31 23:24:48 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.85, 67.36 GiB). Actual usage is 9.03 GiB for consumed memory (weights + non-torch), 2.08 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=59997267047` (55.88 GiB) to fit into requested memory, or `--kv-cache-memory=72318074368` (67.35 GiB) to fully utilize gpu memory. Current kv cache memory in use is 56.26 GiB. | |
| (EngineCore pid=132425) INFO 10-31 23:24:49 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=132425) INFO 10-31 23:24:49 [core.py:395] init engine (profile, create kv cache, warmup model) took 40.43 s (compilation: 1.04 s) | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 5.97it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 10.89it/s] | |
| (EngineCore pid=132425) INFO 10-31 23:24:50 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=132001) INFO 10-31 23:24:51 [hf.py:648] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=132425) WARNING 10-31 23:24:52 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=132001) INFO 10-31 23:25:04 [loggers.py:335] Engine 000: Avg prompt throughput: 19791.3 tokens/s, Avg generation throughput: 36.2 tokens/s, Running: 21 reqs, Waiting: 63 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.2% | |
| (MainProcess pid=132001) INFO 10-31 23:25:14 [loggers.py:335] Engine 000: Avg prompt throughput: 26617.0 tokens/s, Avg generation throughput: 50.4 tokens/s, Running: 17 reqs, Waiting: 457 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.4% | |
| (MainProcess pid=132001) INFO 10-31 23:25:24 [loggers.py:335] Engine 000: Avg prompt throughput: 26816.0 tokens/s, Avg generation throughput: 57.1 tokens/s, Running: 18 reqs, Waiting: 173 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=132001) INFO 10-31 23:25:34 [loggers.py:335] Engine 000: Avg prompt throughput: 26601.0 tokens/s, Avg generation throughput: 57.3 tokens/s, Running: 17 reqs, Waiting: 491 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=132001) INFO 10-31 23:25:44 [loggers.py:335] Engine 000: Avg prompt throughput: 24127.5 tokens/s, Avg generation throughput: 50.8 tokens/s, Running: 18 reqs, Waiting: 14 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=132001) INFO 10-31 23:25:54 [loggers.py:335] Engine 000: Avg prompt throughput: 27360.5 tokens/s, Avg generation throughput: 59.4 tokens/s, Running: 17 reqs, Waiting: 576 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=132001) INFO 10-31 23:26:04 [loggers.py:335] Engine 000: Avg prompt throughput: 27220.7 tokens/s, Avg generation throughput: 57.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=132001) INFO 10-31 23:26:15 [loggers.py:335] Engine 000: Avg prompt throughput: 26046.0 tokens/s, Avg generation throughput: 56.7 tokens/s, Running: 17 reqs, Waiting: 627 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=132001) INFO 10-31 23:26:25 [loggers.py:335] Engine 000: Avg prompt throughput: 27157.5 tokens/s, Avg generation throughput: 57.3 tokens/s, Running: 17 reqs, Waiting: 50 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% | |
| {"saved": "data/jevals/runs/base.json"} | |
| (MainProcess pid=132001) INFO 10-31 23:26:31 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=132425) INFO 10-31 23:26:31 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=132425) INFO 10-31 23:26:31 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=132425) INFO 10-31 23:26:31 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=132425) INFO 10-31 23:26:31 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=133255) INFO 10-31 23:26:49 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'gpu_memory_utilization': 0.85, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'checkpoints/joint_merged'} | |
| (MainProcess pid=133255) INFO 10-31 23:26:49 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM | |
| (MainProcess pid=133255) INFO 10-31 23:26:49 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=133255) INFO 10-31 23:26:49 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=8192. | |
| (MainProcess pid=133255) INFO 10-31 23:26:49 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled | |
| (MainProcess pid=133255) INFO 10-31 23:26:49 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=133604) INFO 10-31 23:26:55 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/joint_merged', speculative_config=None, tokenizer='checkpoints/joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=133604) INFO 10-31 23:26:56 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_070aff801b5c4bc5977aa6c6aa18ff23 backend=nccl | |
| (EngineCore pid=133604) INFO 10-31 23:26:56 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=133604) INFO 10-31 23:26:56 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=133604) INFO 10-31 23:26:58 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=133604) INFO 10-31 23:26:58 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=133604) INFO 10-31 23:26:59 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=133604) INFO 10-31 23:26:59 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=133604) INFO 10-31 23:26:59 [base_loader.py:82] Initializing model took 1.487983 seconds | |
| (EngineCore pid=133604) INFO 10-31 23:26:59 [weight_utils.py:1108] Filesystem type for checkpoints: OVERLAY. Checkpoint size: 7.83 GiB. Available RAM: 135.20 GiB. | |
| (EngineCore pid=133604) INFO 10-31 23:26:59 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (OVERLAY) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.29s/it] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.29s/it] | |
| (EngineCore pid=133604) INFO 10-31 23:27:01 [default_loader.py:501] Loading weights took 1.40 seconds | |
| (EngineCore pid=133604) INFO 10-31 23:27:02 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.775560 seconds | |
| (EngineCore pid=133604) INFO 10-31 23:27:04 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=133604) INFO 10-31 23:27:04 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=133604) INFO 10-31 23:27:04 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=133604) INFO 10-31 23:27:04 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=133604) INFO 10-31 23:27:04 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=133604) INFO 10-31 23:27:05 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=133604) INFO 10-31 23:27:10 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/715398b524/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=133604) INFO 10-31 23:27:10 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.29 s | |
| (EngineCore pid=133604) INFO 10-31 23:27:26 [backends.py:398] [backbone] Compiling a graph for compile range (1, 8192) takes 16.03 s | |
| (EngineCore pid=133604) INFO 10-31 23:27:29 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20441186 bytes total | |
| (EngineCore pid=133604) INFO 10-31 23:27:29 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/9d2fb9b023bc3bde6d919101fe604eed875bdd8c667707451bc6a9a415806035/rank_0_0/model | |
| (EngineCore pid=133604) INFO 10-31 23:27:29 [monitor.py:62] [backbone] torch.compile took 24.26 s in total | |
| (MainProcess pid=134392) INFO 10-31 23:28:55 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'gpu_memory_utilization': 0.85, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'checkpoints/joint_merged'} | |
| (MainProcess pid=134392) INFO 10-31 23:28:55 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM | |
| (MainProcess pid=134392) INFO 10-31 23:28:55 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=134392) INFO 10-31 23:28:55 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=8192. | |
| (MainProcess pid=134392) INFO 10-31 23:28:55 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled | |
| (MainProcess pid=134392) INFO 10-31 23:28:55 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=134669) INFO 10-31 23:29:01 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/joint_merged', speculative_config=None, tokenizer='checkpoints/joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=134669) INFO 10-31 23:29:02 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_7a845a6103ff42e6839f50c41ab62501 backend=nccl | |
| (EngineCore pid=134669) INFO 10-31 23:29:02 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=134669) INFO 10-31 23:29:02 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=134669) INFO 10-31 23:29:04 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=134669) INFO 10-31 23:29:04 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=134669) INFO 10-31 23:29:05 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=134669) INFO 10-31 23:29:05 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=134669) INFO 10-31 23:29:05 [base_loader.py:82] Initializing model took 1.540498 seconds | |
| (EngineCore pid=134669) INFO 10-31 23:29:05 [weight_utils.py:1108] Filesystem type for checkpoints: OVERLAY. Checkpoint size: 7.83 GiB. Available RAM: 135.12 GiB. | |
| (EngineCore pid=134669) INFO 10-31 23:29:05 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (OVERLAY) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.27s/it] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.27s/it] | |
| (EngineCore pid=134669) INFO 10-31 23:29:07 [default_loader.py:501] Loading weights took 1.37 seconds | |
| (EngineCore pid=134669) INFO 10-31 23:29:08 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.709725 seconds | |
| (EngineCore pid=134669) INFO 10-31 23:29:10 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=134669) INFO 10-31 23:29:10 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=134669) INFO 10-31 23:29:10 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=134669) INFO 10-31 23:29:10 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=134669) INFO 10-31 23:29:10 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=134669) INFO 10-31 23:29:12 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=12 num_submods=33 | |
| (EngineCore pid=134669) INFO 10-31 23:29:12 [decorators.py:319] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/9d2fb9b023bc3bde6d919101fe604eed875bdd8c667707451bc6a9a415806035/rank_0_0/model | |
| (EngineCore pid=134669) INFO 10-31 23:29:12 [monitor.py:62] [backbone] torch.compile took 0.98 s in total | |
| (EngineCore pid=134669) INFO 10-31 23:29:55 [monitor.py:91] [backbone] Initial profiling/warmup run took 42.89 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:01<00:54, 1.10s/it] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:47, 1.02it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:35, 1.36it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:02<00:28, 1.64it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:25, 1.82it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:03<00:22, 2.01it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:20, 2.19it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:04<00:20, 2.14it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:19, 2.19it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:05<00:17, 2.31it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:05<00:16, 2.49it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:15, 2.57it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:06<00:14, 2.55it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:06<00:13, 2.67it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:12, 2.82it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:07<00:12, 2.90it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:08<00:18, 1.83it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:15, 2.19it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:08<00:06, 4.41it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:08<00:03, 6.79it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:09<00:03, 7.66it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:09<00:02, 9.36it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:09<00:01, 11.80it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:09<00:01, 11.19it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:09<00:01, 13.51it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:09<00:00, 14.64it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:10<00:00, 12.84it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:10<00:00, 15.00it/s] Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:10<00:00, 15.56it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:10<00:00, 14.68it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:10<00:00, 16.38it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 4.70it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 27.73it/s] | |
| (EngineCore pid=134669) INFO 10-31 23:30:08 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.40 GiB | |
| (EngineCore pid=134669) INFO 10-31 23:30:09 [gpu_worker.py:759] Available KV cache memory: 55.62 GiB | |
| (EngineCore pid=134669) INFO 10-31 23:30:09 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.8500 is equivalent to --gpu-memory-utilization=0.8440 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.8560. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=134669) INFO 10-31 23:30:09 [kv_cache_utils.py:2327] GPU KV cache size: 1,638,874 tokens, Maximum concurrency for 32,768 tokens per request: 50.01x | |
| (EngineCore pid=134669) INFO 10-31 23:30:09 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=134669) INFO 10-31 23:30:09 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=134669) INFO 10-31 23:30:09 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=134669) INFO 10-31 23:30:09 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=134669) INFO 10-31 23:30:09 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=134669) INFO 10-31 23:30:10 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:37, 1.33it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:03<01:37, 1.98s/it] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:04<01:08, 1.43s/it] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:05<00:53, 1.15s/it] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:05<00:44, 1.03it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:06<00:38, 1.18it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:06<00:32, 1.34it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:07<00:28, 1.49it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:07<00:25, 1.66it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:08<00:22, 1.82it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:08<00:20, 1.92it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:09<00:19, 2.04it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:09<00:16, 2.30it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:09<00:15, 2.44it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:10<00:13, 2.63it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:10<00:11, 3.17it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:10<00:05, 6.09it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:10<00:04, 6.85it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:10<00:02, 9.69it/s] Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:10<00:01, 12.17it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:11<00:01, 11.07it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:11<00:01, 13.36it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:11<00:01, 14.60it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:11<00:01, 13.61it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:11<00:00, 15.58it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:11<00:00, 17.18it/s] Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:12<00:00, 13.74it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:12<00:00, 15.64it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:12<00:00, 17.19it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:12<00:00, 4.10it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 16.84it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 21.10it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 23.50it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:00, 24.64it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:01, 18.73it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 20.72it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 22.55it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 23.92it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 25.00it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 26.11it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 26.60it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 23.82it/s] | |
| (EngineCore pid=134669) INFO 10-31 23:30:43 [model_runner.py:1127] Graph capturing finished in 15 secs, took 0.24 GiB | |
| (EngineCore pid=134669) INFO 10-31 23:30:43 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.23 GiB (97.6%). | |
| (EngineCore pid=134669) INFO 10-31 23:30:43 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.85, 67.36 GiB). Actual usage is 10.52 GiB for consumed memory (weights + non-torch), 1.22 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=59305185383` (55.23 GiB) to fit into requested memory, or `--kv-cache-memory=71625992704` (66.71 GiB) to fully utilize gpu memory. Current kv cache memory in use is 55.62 GiB. | |
| (EngineCore pid=134669) INFO 10-31 23:30:44 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=134669) INFO 10-31 23:30:44 [core.py:395] init engine (profile, create kv cache, warmup model) took 94.77 s (compilation: 0.98 s) | |
| (EngineCore pid=134669) INFO 10-31 23:30:45 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=134392) INFO 10-31 23:30:46 [hf.py:648] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=134669) WARNING 10-31 23:30:46 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=134392) INFO 10-31 23:30:57 [loggers.py:335] Engine 000: Avg prompt throughput: 21976.5 tokens/s, Avg generation throughput: 40.2 tokens/s, Running: 21 reqs, Waiting: 63 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.2% | |
| (MainProcess pid=134392) INFO 10-31 23:31:07 [loggers.py:335] Engine 000: Avg prompt throughput: 25866.1 tokens/s, Avg generation throughput: 48.8 tokens/s, Running: 17 reqs, Waiting: 474 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.4% | |
| (MainProcess pid=134392) INFO 10-31 23:31:18 [loggers.py:335] Engine 000: Avg prompt throughput: 26609.2 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 18 reqs, Waiting: 191 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=134392) INFO 10-31 23:31:28 [loggers.py:335] Engine 000: Avg prompt throughput: 26340.6 tokens/s, Avg generation throughput: 56.8 tokens/s, Running: 17 reqs, Waiting: 508 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=134392) INFO 10-31 23:31:38 [loggers.py:335] Engine 000: Avg prompt throughput: 26496.9 tokens/s, Avg generation throughput: 56.0 tokens/s, Running: 18 reqs, Waiting: 245 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=134392) INFO 10-31 23:31:48 [loggers.py:335] Engine 000: Avg prompt throughput: 26210.0 tokens/s, Avg generation throughput: 56.8 tokens/s, Running: 17 reqs, Waiting: 576 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=134392) INFO 10-31 23:31:58 [loggers.py:335] Engine 000: Avg prompt throughput: 26870.5 tokens/s, Avg generation throughput: 56.7 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=134392) INFO 10-31 23:32:08 [loggers.py:335] Engine 000: Avg prompt throughput: 25777.3 tokens/s, Avg generation throughput: 56.1 tokens/s, Running: 17 reqs, Waiting: 627 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=134392) INFO 10-31 23:32:19 [loggers.py:335] Engine 000: Avg prompt throughput: 26846.7 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 17 reqs, Waiting: 50 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% | |
| {"saved": "data/jevals/runs/v1_joint.json"} | |
| (MainProcess pid=134392) INFO 10-31 23:32:25 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=134669) INFO 10-31 23:32:25 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=134669) INFO 10-31 23:32:25 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=134669) INFO 10-31 23:32:25 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=134669) INFO 10-31 23:32:25 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=134392) WARNING 10-31 23:32:30 [utils.py:648] [shutdown] Process manager: force killing remaining processes count=1 | |
| (MainProcess pid=134392) WARNING 10-31 23:32:30 [utils.py:653] [shutdown] Process manager: force killing remaining process EngineCore pid 134669 | |
| (MainProcess pid=135795) INFO 10-31 23:32:45 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'runs/v2/checkpoints/v2_joint_merged'} | |
| (MainProcess pid=135795) INFO 10-31 23:32:45 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM | |
| (MainProcess pid=135795) INFO 10-31 23:32:45 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=135795) INFO 10-31 23:32:45 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=135795) INFO 10-31 23:32:45 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled | |
| (MainProcess pid=135795) INFO 10-31 23:32:45 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=136245) INFO 10-31 23:32:51 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='runs/v2/checkpoints/v2_joint_merged', speculative_config=None, tokenizer='runs/v2/checkpoints/v2_joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=runs/v2/checkpoints/v2_joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=136245) INFO 10-31 23:32:52 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_691db67431d34ef4a59f9170364cdf4a backend=nccl | |
| (EngineCore pid=136245) INFO 10-31 23:32:52 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=136245) INFO 10-31 23:32:52 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=136245) INFO 10-31 23:32:54 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=136245) INFO 10-31 23:32:54 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=136245) INFO 10-31 23:32:55 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=136245) INFO 10-31 23:32:55 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=136245) INFO 10-31 23:32:55 [base_loader.py:82] Initializing model took 1.620193 seconds | |
| (EngineCore pid=136245) INFO 10-31 23:32:55 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 135.06 GiB. | |
| (EngineCore pid=136245) INFO 10-31 23:32:55 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.17it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.17it/s] | |
| (EngineCore pid=136245) INFO 10-31 23:32:56 [default_loader.py:501] Loading weights took 0.89 seconds | |
| (EngineCore pid=136245) INFO 10-31 23:32:58 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.320535 seconds | |
| (EngineCore pid=136245) INFO 10-31 23:32:58 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=136245) INFO 10-31 23:32:58 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=136245) INFO 10-31 23:32:58 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=136245) INFO 10-31 23:32:58 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=136245) INFO 10-31 23:32:58 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=136245) INFO 10-31 23:32:59 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=136245) INFO 10-31 23:33:04 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/3314745679/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=136245) INFO 10-31 23:33:04 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.29 s | |
| (EngineCore pid=136245) INFO 10-31 23:33:20 [backends.py:398] [backbone] Compiling a graph for compile range (1, 32768) takes 16.02 s | |
| (EngineCore pid=136245) INFO 10-31 23:33:23 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20354505 bytes total | |
| (EngineCore pid=136245) INFO 10-31 23:33:23 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/56bd496484f38fb25a287ea46eefbaf2a424ccc54f7b3760d02475d12ea1e618/rank_0_0/model | |
| (EngineCore pid=136245) INFO 10-31 23:33:23 [monitor.py:62] [backbone] torch.compile took 24.28 s in total | |
| (EngineCore pid=136245) INFO 10-31 23:34:50 [monitor.py:91] [backbone] Initial profiling/warmup run took 87.36 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:46, 1.07it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:40, 1.20it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:28, 1.66it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:02<00:23, 2.04it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:02<00:22, 2.03it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:03<00:23, 1.89it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:03<00:23, 1.85it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:04<00:22, 1.91it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:20, 2.01it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:05<00:18, 2.16it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:05<00:16, 2.36it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:05<00:14, 2.62it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:06<00:13, 2.82it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:06<00:12, 3.05it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:06<00:10, 3.34it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:06<00:09, 3.67it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:08<00:17, 1.94it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:14, 2.28it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:08<00:08, 3.85it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:08<00:04, 6.48it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:08<00:03, 8.26it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:02, 10.90it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:08<00:01, 12.48it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:09<00:01, 14.49it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:09<00:01, 15.62it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:09<00:00, 16.50it/s] Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:09<00:00, 17.33it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:09<00:00, 18.41it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:09<00:00, 18.72it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:09<00:00, 18.45it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:09<00:00, 18.82it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 13.86it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 5.05it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 25.74it/s] | |
| (EngineCore pid=136245) INFO 10-31 23:35:04 [model_runner.py:1127] Graph capturing finished in 11 secs, took 0.40 GiB | |
| (EngineCore pid=136245) INFO 10-31 23:35:04 [gpu_worker.py:759] Available KV cache memory: 58.91 GiB | |
| (EngineCore pid=136245) INFO 10-31 23:35:04 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9140 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9260. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=136245) INFO 10-31 23:35:04 [kv_cache_utils.py:2327] GPU KV cache size: 1,736,229 tokens, Maximum concurrency for 32,768 tokens per request: 52.99x | |
| (EngineCore pid=136245) INFO 10-31 23:35:04 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=136245) INFO 10-31 23:35:04 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=136245) INFO 10-31 23:35:05 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=136245) INFO 10-31 23:35:05 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=136245) INFO 10-31 23:35:05 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=136245) INFO 10-31 23:35:06 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:42, 1.18it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:40, 1.21it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:38, 1.23it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:36, 1.30it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:33, 1.36it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:30, 1.46it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:28, 1.56it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:25, 1.68it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:05<00:22, 1.83it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:20, 2.01it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:18, 2.22it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:15, 2.50it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:14, 2.71it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:12, 2.95it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:10, 3.32it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:07<00:08, 3.93it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:07<00:04, 7.29it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:08<00:02, 10.37it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:08<00:01, 13.00it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:01, 15.14it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:08<00:01, 16.63it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:08<00:00, 17.74it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:08<00:00, 18.40it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:08<00:00, 19.17it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:09<00:00, 18.34it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:09<00:00, 18.23it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:09<00:00, 18.09it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:09<00:00, 18.99it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:09<00:00, 5.33it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 28.78it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 28.01it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:00, 27.83it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:00, 27.52it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 27.47it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 27.34it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 27.63it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:00<00:00, 27.43it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:00<00:00, 27.80it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 28.20it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 28.42it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 27.99it/s] | |
| (EngineCore pid=136245) INFO 10-31 23:35:37 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.24 GiB | |
| (EngineCore pid=136245) INFO 10-31 23:35:37 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (99.2%). | |
| (EngineCore pid=136245) INFO 10-31 23:35:37 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 10.52 GiB for consumed memory (weights + non-torch), 3.47 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=62845904098` (58.53 GiB) to fit into requested memory, or `--kv-cache-memory=69210073600` (64.46 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.91 GiB. | |
| (EngineCore pid=136245) INFO 10-31 23:35:38 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=136245) INFO 10-31 23:35:38 [core.py:395] init engine (profile, create kv cache, warmup model) took 160.40 s (compilation: 24.28 s) | |
| (EngineCore pid=136245) INFO 10-31 23:35:38 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=135795) INFO 10-31 23:35:40 [hf.py:648] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=136245) WARNING 10-31 23:35:40 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=135795) INFO 10-31 23:35:52 [loggers.py:335] Engine 000: Avg prompt throughput: 21082.0 tokens/s, Avg generation throughput: 38.6 tokens/s, Running: 34 reqs, Waiting: 46 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 0.2% | |
| (MainProcess pid=135795) INFO 10-31 23:36:02 [loggers.py:335] Engine 000: Avg prompt throughput: 27811.5 tokens/s, Avg generation throughput: 53.1 tokens/s, Running: 68 reqs, Waiting: 349 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.4% | |
| (MainProcess pid=135795) INFO 10-31 23:36:13 [loggers.py:335] Engine 000: Avg prompt throughput: 27214.3 tokens/s, Avg generation throughput: 58.3 tokens/s, Running: 72 reqs, Waiting: 50 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=135795) INFO 10-31 23:36:24 [loggers.py:335] Engine 000: Avg prompt throughput: 26795.5 tokens/s, Avg generation throughput: 57.2 tokens/s, Running: 68 reqs, Waiting: 322 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=135795) INFO 10-31 23:36:35 [loggers.py:335] Engine 000: Avg prompt throughput: 26920.9 tokens/s, Avg generation throughput: 57.9 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=135795) INFO 10-31 23:36:46 [loggers.py:335] Engine 000: Avg prompt throughput: 26563.8 tokens/s, Avg generation throughput: 56.3 tokens/s, Running: 70 reqs, Waiting: 252 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=135795) INFO 10-31 23:36:56 [loggers.py:335] Engine 000: Avg prompt throughput: 26923.1 tokens/s, Avg generation throughput: 58.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=135795) INFO 10-31 23:37:07 [loggers.py:335] Engine 000: Avg prompt throughput: 26516.0 tokens/s, Avg generation throughput: 56.0 tokens/s, Running: 70 reqs, Waiting: 252 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=135795) INFO 10-31 23:37:17 [loggers.py:335] Engine 000: Avg prompt throughput: 26932.6 tokens/s, Avg generation throughput: 58.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% | |
| {"saved": "data/jevals/runs/v2_joint.json"} | |
| (MainProcess pid=135795) INFO 10-31 23:37:18 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=136245) INFO 10-31 23:37:18 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=136245) INFO 10-31 23:37:18 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=136245) INFO 10-31 23:37:18 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=136245) INFO 10-31 23:37:18 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=137336) INFO 10-31 23:37:36 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'runs/v2/checkpoints/rl60_merged'} | |
| (MainProcess pid=137336) INFO 10-31 23:37:36 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM | |
| (MainProcess pid=137336) INFO 10-31 23:37:36 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=137336) INFO 10-31 23:37:36 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=137336) INFO 10-31 23:37:36 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled | |
| (MainProcess pid=137336) INFO 10-31 23:37:36 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=137604) INFO 10-31 23:37:42 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='runs/v2/checkpoints/rl60_merged', speculative_config=None, tokenizer='runs/v2/checkpoints/rl60_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=runs/v2/checkpoints/rl60_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=137604) INFO 10-31 23:37:44 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_d3f42a4792384ae3a1853ffd22ca234e backend=nccl | |
| (EngineCore pid=137604) INFO 10-31 23:37:44 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=137604) INFO 10-31 23:37:44 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=137604) INFO 10-31 23:37:45 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=137604) INFO 10-31 23:37:45 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=137604) INFO 10-31 23:37:47 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=137604) INFO 10-31 23:37:47 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=137604) INFO 10-31 23:37:47 [base_loader.py:82] Initializing model took 1.600326 seconds | |
| (EngineCore pid=137604) INFO 10-31 23:37:47 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 134.96 GiB. | |
| (EngineCore pid=137604) INFO 10-31 23:37:47 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.19s/it] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.19s/it] | |
| (EngineCore pid=137604) INFO 10-31 23:37:48 [default_loader.py:501] Loading weights took 1.22 seconds | |
| (EngineCore pid=137604) INFO 10-31 23:37:50 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.586027 seconds | |
| (EngineCore pid=137604) INFO 10-31 23:37:50 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=137604) INFO 10-31 23:37:50 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=137604) INFO 10-31 23:37:50 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=137604) INFO 10-31 23:37:50 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=137604) INFO 10-31 23:37:50 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=137604) INFO 10-31 23:37:50 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=137604) INFO 10-31 23:37:56 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/9251680859/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=137604) INFO 10-31 23:37:56 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.29 s | |
| (EngineCore pid=137604) INFO 10-31 23:38:13 [backends.py:398] [backbone] Compiling a graph for compile range (1, 32768) takes 17.44 s | |
| (EngineCore pid=137604) INFO 10-31 23:38:16 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20354505 bytes total | |
| (EngineCore pid=137604) INFO 10-31 23:38:16 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/d0b0f9304b6ae0d7059ba6c9f00d7875425c6aade26d61e254d229a76ead35b3/rank_0_0/model | |
| (EngineCore pid=137604) INFO 10-31 23:38:16 [monitor.py:62] [backbone] torch.compile took 25.77 s in total | |
| (EngineCore pid=137604) INFO 10-31 23:39:44 [monitor.py:91] [backbone] Initial profiling/warmup run took 87.96 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:47, 1.04it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:41, 1.18it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:29, 1.61it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:02<00:23, 1.98it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:02<00:21, 2.09it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:03<00:23, 1.93it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:03<00:23, 1.89it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:04<00:22, 1.93it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:20, 2.01it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:05<00:18, 2.16it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:05<00:16, 2.37it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:05<00:14, 2.61it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:06<00:13, 2.79it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:06<00:12, 3.02it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:06<00:11, 3.26it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:06<00:09, 3.58it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:08<00:17, 1.89it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:14, 2.26it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:08<00:08, 3.82it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:08<00:04, 6.44it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:08<00:03, 8.18it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:02, 10.88it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:08<00:01, 13.12it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:09<00:01, 14.38it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:09<00:00, 16.18it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:09<00:00, 16.67it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:09<00:00, 17.86it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:09<00:00, 18.80it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:09<00:00, 19.47it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:09<00:00, 19.83it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 5.04it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 26.15it/s] | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [model_runner.py:1127] Graph capturing finished in 11 secs, took 0.40 GiB | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [gpu_worker.py:759] Available KV cache memory: 58.91 GiB | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9140 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9260. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [kv_cache_utils.py:2327] GPU KV cache size: 1,736,229 tokens, Maximum concurrency for 32,768 tokens per request: 52.99x | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=137604) INFO 10-31 23:39:58 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=137604) INFO 10-31 23:40:00 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:42, 1.18it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:40, 1.21it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:38, 1.23it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:36, 1.30it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:33, 1.37it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:30, 1.45it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:28, 1.56it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:25, 1.68it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:05<00:22, 1.83it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:20, 1.99it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:17, 2.23it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:15, 2.49it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:14, 2.68it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:12, 2.94it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:10, 3.31it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:07<00:08, 3.91it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:08<00:04, 7.23it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:08<00:02, 10.31it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:08<00:02, 12.97it/s] Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:08<00:01, 14.24it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:08<00:01, 15.40it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:08<00:01, 15.07it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:08<00:01, 15.02it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:08<00:00, 16.93it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:09<00:00, 18.09it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:09<00:00, 19.09it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:09<00:00, 19.84it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:09<00:00, 20.43it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:09<00:00, 5.31it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 27.03it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 26.83it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:00, 26.45it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:00, 26.43it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 26.46it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 26.49it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 26.61it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:00<00:00, 26.53it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 26.80it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 26.69it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 27.02it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 26.81it/s] | |
| (EngineCore pid=137604) INFO 10-31 23:40:30 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.24 GiB | |
| (EngineCore pid=137604) INFO 10-31 23:40:30 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (99.2%). | |
| (EngineCore pid=137604) INFO 10-31 23:40:30 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 10.52 GiB for consumed memory (weights + non-torch), 3.47 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=62845904098` (58.53 GiB) to fit into requested memory, or `--kv-cache-memory=69210073600` (64.46 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.91 GiB. | |
| (EngineCore pid=137604) INFO 10-31 23:40:31 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=137604) INFO 10-31 23:40:32 [core.py:395] init engine (profile, create kv cache, warmup model) took 161.95 s (compilation: 25.77 s) | |
| (EngineCore pid=137604) INFO 10-31 23:40:32 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=137336) INFO 10-31 23:40:33 [hf.py:648] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=137604) WARNING 10-31 23:40:34 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=137336) INFO 10-31 23:40:45 [loggers.py:335] Engine 000: Avg prompt throughput: 21313.9 tokens/s, Avg generation throughput: 39.1 tokens/s, Running: 34 reqs, Waiting: 46 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 0.2% | |
| (MainProcess pid=137336) INFO 10-31 23:40:55 [loggers.py:335] Engine 000: Avg prompt throughput: 28026.0 tokens/s, Avg generation throughput: 53.5 tokens/s, Running: 68 reqs, Waiting: 349 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.4% | |
| (MainProcess pid=137336) INFO 10-31 23:41:06 [loggers.py:335] Engine 000: Avg prompt throughput: 27108.6 tokens/s, Avg generation throughput: 58.0 tokens/s, Running: 73 reqs, Waiting: 50 reqs, GPU KV cache usage: 8.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=137336) INFO 10-31 23:41:17 [loggers.py:335] Engine 000: Avg prompt throughput: 26826.8 tokens/s, Avg generation throughput: 57.2 tokens/s, Running: 68 reqs, Waiting: 315 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=137336) INFO 10-31 23:41:28 [loggers.py:335] Engine 000: Avg prompt throughput: 27160.9 tokens/s, Avg generation throughput: 58.5 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=137336) INFO 10-31 23:41:39 [loggers.py:335] Engine 000: Avg prompt throughput: 26747.2 tokens/s, Avg generation throughput: 56.7 tokens/s, Running: 70 reqs, Waiting: 246 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=137336) INFO 10-31 23:41:49 [loggers.py:335] Engine 000: Avg prompt throughput: 27138.8 tokens/s, Avg generation throughput: 58.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=137336) INFO 10-31 23:42:00 [loggers.py:335] Engine 000: Avg prompt throughput: 26682.5 tokens/s, Avg generation throughput: 56.3 tokens/s, Running: 70 reqs, Waiting: 246 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=137336) INFO 10-31 23:42:10 [loggers.py:335] Engine 000: Avg prompt throughput: 27101.6 tokens/s, Avg generation throughput: 58.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% | |
| {"saved": "data/jevals/runs/rl60.json"} | |
| (MainProcess pid=137336) INFO 10-31 23:42:10 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=137604) INFO 10-31 23:42:10 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=137604) INFO 10-31 23:42:10 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=137604) INFO 10-31 23:42:10 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=137604) INFO 10-31 23:42:10 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=138633) INFO 10-31 23:42:32 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 384, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'runs/v2/checkpoints/rl60_merged'} | |
| (MainProcess pid=138633) INFO 10-31 23:42:32 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM | |
| (MainProcess pid=138633) INFO 10-31 23:42:32 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=138633) INFO 10-31 23:42:32 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=138633) INFO 10-31 23:42:32 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled | |
| (MainProcess pid=138633) INFO 10-31 23:42:32 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=138904) INFO 10-31 23:42:38 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='runs/v2/checkpoints/rl60_merged', speculative_config=None, tokenizer='runs/v2/checkpoints/rl60_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=runs/v2/checkpoints/rl60_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=138904) INFO 10-31 23:42:39 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_bf060d3271604e5d85ab709a502c037f backend=nccl | |
| (EngineCore pid=138904) INFO 10-31 23:42:39 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=138904) INFO 10-31 23:42:39 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=138904) INFO 10-31 23:42:40 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=138904) INFO 10-31 23:42:40 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=138904) INFO 10-31 23:42:42 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=138904) INFO 10-31 23:42:42 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=138904) INFO 10-31 23:42:42 [base_loader.py:82] Initializing model took 1.575504 seconds | |
| (EngineCore pid=138904) INFO 10-31 23:42:42 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 134.72 GiB. | |
| (EngineCore pid=138904) INFO 10-31 23:42:42 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.06s/it] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.06s/it] | |
| (EngineCore pid=138904) INFO 10-31 23:42:43 [default_loader.py:501] Loading weights took 1.09 seconds | |
| (EngineCore pid=138904) INFO 10-31 23:42:45 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.457360 seconds | |
| (EngineCore pid=138904) INFO 10-31 23:42:45 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=138904) INFO 10-31 23:42:45 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=138904) INFO 10-31 23:42:45 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=138904) INFO 10-31 23:42:45 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=138904) INFO 10-31 23:42:45 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=138904) INFO 10-31 23:42:46 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=138904) INFO 10-31 23:42:51 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/a9acc19357/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=138904) INFO 10-31 23:42:51 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.38 s | |
| (EngineCore pid=138904) INFO 10-31 23:43:08 [backends.py:398] [backbone] Compiling a graph for compile range (1, 32768) takes 17.35 s | |
| (EngineCore pid=138904) INFO 10-31 23:43:11 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20355017 bytes total | |
| (EngineCore pid=138904) INFO 10-31 23:43:11 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/b4062aebc42f7fd3ef9b60e99d390d27b84130defa71ea9f4dbde346c5565588/rank_0_0/model | |
| (EngineCore pid=138904) INFO 10-31 23:43:11 [monitor.py:62] [backbone] torch.compile took 25.83 s in total | |
| (EngineCore pid=138904) INFO 10-31 23:44:40 [monitor.py:91] [backbone] Initial profiling/warmup run took 88.36 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:01<01:04, 1.30s/it] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:34, 1.44it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:01<00:25, 1.87it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:02<00:19, 2.46it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:02<00:17, 2.57it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:02<00:16, 2.70it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:02<00:13, 3.24it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:03<00:11, 3.86it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:21, 1.93it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:04<00:11, 3.36it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:04<00:06, 5.78it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:04<00:05, 6.95it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:04<00:04, 7.15it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:05<00:03, 8.89it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:05<00:02, 9.87it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:05<00:02, 12.55it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:05<00:01, 14.63it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:05<00:01, 13.47it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:05<00:01, 15.23it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:05<00:00, 16.54it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:06<00:00, 15.33it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:06<00:00, 16.28it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:06<00:00, 16.39it/s] Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:06<00:00, 15.87it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:06<00:00, 17.37it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:06<00:00, 13.47it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:06<00:00, 7.30it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 23.52it/s] | |
| (EngineCore pid=138904) INFO 10-31 23:44:50 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.42 GiB | |
| (EngineCore pid=138904) INFO 10-31 23:44:51 [gpu_worker.py:759] Available KV cache memory: 58.78 GiB | |
| (EngineCore pid=138904) INFO 10-31 23:44:51 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9138 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9262. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=138904) INFO 10-31 23:44:51 [kv_cache_utils.py:2327] GPU KV cache size: 1,731,955 tokens, Maximum concurrency for 32,768 tokens per request: 52.86x | |
| (EngineCore pid=138904) INFO 10-31 23:44:51 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=138904) INFO 10-31 23:44:51 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=138904) INFO 10-31 23:44:51 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=138904) INFO 10-31 23:44:51 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=138904) INFO 10-31 23:44:51 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=138904) INFO 10-31 23:44:52 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:16, 2.96it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:17, 2.74it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:01<00:16, 2.89it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:01<00:14, 3.15it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:01<00:13, 3.32it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:01<00:12, 3.56it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:02<00:11, 3.93it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:02<00:09, 4.62it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:02<00:05, 7.25it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:02<00:03, 10.84it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:02<00:02, 13.72it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:02<00:02, 15.99it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:02<00:01, 17.66it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:02<00:01, 18.93it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:03<00:01, 19.83it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:03<00:00, 20.13it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:03<00:00, 20.32it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:03<00:00, 20.77it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:03<00:00, 20.65it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:03<00:00, 18.39it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:04<00:00, 19.34it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:04<00:00, 20.05it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:04<00:00, 11.88it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/43 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 7%|▋ | 3/43 [00:00<00:01, 24.87it/s] Capturing CUDA graphs (FULL): 14%|█▍ | 6/43 [00:00<00:01, 24.97it/s] Capturing CUDA graphs (FULL): 21%|██ | 9/43 [00:00<00:01, 25.14it/s] Capturing CUDA graphs (FULL): 28%|██▊ | 12/43 [00:00<00:01, 21.31it/s] Capturing CUDA graphs (FULL): 35%|███▍ | 15/43 [00:00<00:01, 23.19it/s] Capturing CUDA graphs (FULL): 42%|████▏ | 18/43 [00:00<00:01, 24.40it/s] Capturing CUDA graphs (FULL): 49%|████▉ | 21/43 [00:00<00:00, 25.19it/s] Capturing CUDA graphs (FULL): 56%|█████▌ | 24/43 [00:01<00:00, 23.25it/s] Capturing CUDA graphs (FULL): 63%|██████▎ | 27/43 [00:01<00:00, 24.43it/s] Capturing CUDA graphs (FULL): 70%|██████▉ | 30/43 [00:01<00:00, 25.46it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 33/43 [00:01<00:00, 25.83it/s] Capturing CUDA graphs (FULL): 84%|████████▎ | 36/43 [00:01<00:00, 22.54it/s] Capturing CUDA graphs (FULL): 91%|█████████ | 39/43 [00:01<00:00, 23.74it/s] Capturing CUDA graphs (FULL): 98%|█████████▊| 42/43 [00:01<00:00, 24.91it/s] Capturing CUDA graphs (FULL): 100%|██████████| 43/43 [00:01<00:00, 24.32it/s] | |
| (EngineCore pid=138904) INFO 10-31 23:45:18 [model_runner.py:1127] Graph capturing finished in 7 secs, took 0.29 GiB | |
| (EngineCore pid=138904) INFO 10-31 23:45:18 [gpu_worker.py:971] CUDA graph pool memory: 0.29 GiB (actual), 0.49 GiB (estimated), difference: 0.2 GiB (66.7%). | |
| (EngineCore pid=138904) INFO 10-31 23:45:18 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 10.65 GiB for consumed memory (weights + non-torch), 3.49 GiB for peak activation, and 0.29 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=62638283490` (58.34 GiB) to fit into requested memory, or `--kv-cache-memory=69002452992` (64.26 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.78 GiB. | |
| (EngineCore pid=138904) INFO 10-31 23:45:18 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=138904) INFO 10-31 23:45:19 [core.py:395] init engine (profile, create kv cache, warmup model) took 154.28 s (compilation: 25.83 s) | |
| (EngineCore pid=138904) INFO 10-31 23:45:19 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=138633) INFO 10-31 23:45:21 [hf.py:648] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=138904) WARNING 10-31 23:45:21 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=138904) WARNING 10-31 23:45:22 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=138904) WARNING 10-31 23:45:23 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=138633) INFO 10-31 23:45:33 [loggers.py:335] Engine 000: Avg prompt throughput: 9624.6 tokens/s, Avg generation throughput: 3025.8 tokens/s, Running: 300 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:45:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7230.9 tokens/s, Running: 291 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:45:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6737.3 tokens/s, Running: 161 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:46:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5436.6 tokens/s, Running: 72 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:46:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3004.8 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:46:24 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1378.0 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:46:34 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 815.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:46:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 400.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:46:54 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 256.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=138633) INFO 10-31 23:47:04 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 207.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0% | |
| (EngineCore pid=138904) WARNING 10-31 23:47:09 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=138633) INFO 10-31 23:47:15 [loggers.py:335] Engine 000: Avg prompt throughput: 12380.5 tokens/s, Avg generation throughput: 64.3 tokens/s, Running: 47 reqs, Waiting: 157 reqs, GPU KV cache usage: 10.0%, Prefix cache hit rate: 4.8% | |
| {"board": "pubmedqa", "closed": 300, "of": 300, "tokens_median": 727.5} | |
| (MainProcess pid=138633) INFO 10-31 23:47:25 [loggers.py:335] Engine 000: Avg prompt throughput: 28892.6 tokens/s, Avg generation throughput: 25.8 tokens/s, Running: 51 reqs, Waiting: 249 reqs, GPU KV cache usage: 9.4%, Prefix cache hit rate: 5.9% | |
| (MainProcess pid=138633) INFO 10-31 23:47:35 [loggers.py:335] Engine 000: Avg prompt throughput: 18112.6 tokens/s, Avg generation throughput: 2377.3 tokens/s, Running: 300 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.1%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:47:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7043.7 tokens/s, Running: 298 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.3%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:47:55 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6687.7 tokens/s, Running: 258 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.6%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:48:05 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6408.0 tokens/s, Running: 193 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.2%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:48:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5662.7 tokens/s, Running: 155 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.3%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:48:25 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5182.7 tokens/s, Running: 144 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.6%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:48:35 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4829.5 tokens/s, Running: 134 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.2%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:48:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4901.3 tokens/s, Running: 118 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.0%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:48:55 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4606.2 tokens/s, Running: 100 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:49:05 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4114.0 tokens/s, Running: 77 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:49:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3585.5 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.9%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:49:25 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3137.0 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:49:35 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2536.1 tokens/s, Running: 24 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:49:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1551.5 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:49:55 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 747.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:50:05 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 528.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:50:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 354.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:50:25 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 351.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:50:35 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 347.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:50:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 344.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:50:55 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 341.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:51:05 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 338.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:51:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 335.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:51:25 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 332.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:51:35 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 329.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:51:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 327.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:51:55 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 324.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:52:05 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 321.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:52:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 319.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:52:25 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 316.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:52:35 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 314.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:52:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 312.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:52:55 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 309.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:53:05 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 307.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:53:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 305.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:53:25 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 303.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:53:35 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 300.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:53:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 298.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.4%, Prefix cache hit rate: 4.8% | |
| (MainProcess pid=138633) INFO 10-31 23:53:56 [loggers.py:335] Engine 000: Avg prompt throughput: 18358.2 tokens/s, Avg generation throughput: 39.6 tokens/s, Running: 32 reqs, Waiting: 184 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 8.1% | |
| (MainProcess pid=138633) INFO 10-31 23:54:07 [loggers.py:335] Engine 000: Avg prompt throughput: 26177.3 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 25 reqs, Waiting: 54 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 8.8% | |
| {"board": "helpsteer2", "closed": 297, "of": 300, "tokens_median": 1271.0} | |
| (MainProcess pid=138633) INFO 10-31 23:54:18 [loggers.py:335] Engine 000: Avg prompt throughput: 29070.8 tokens/s, Avg generation throughput: 26.7 tokens/s, Running: 213 reqs, Waiting: 87 reqs, GPU KV cache usage: 23.4%, Prefix cache hit rate: 7.6% | |
| (MainProcess pid=138633) INFO 10-31 23:54:28 [loggers.py:335] Engine 000: Avg prompt throughput: 7109.5 tokens/s, Avg generation throughput: 5489.1 tokens/s, Running: 300 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.1%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:54:38 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7158.7 tokens/s, Running: 274 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.6%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:54:48 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6691.4 tokens/s, Running: 159 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.2%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:54:58 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5594.6 tokens/s, Running: 96 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:55:08 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4853.5 tokens/s, Running: 76 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:55:18 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3948.9 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.6%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:55:28 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3159.4 tokens/s, Running: 33 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.1%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:55:38 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2230.8 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:55:48 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1369.3 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:55:58 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 674.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:56:08 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 304.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:56:18 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 182.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:56:28 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 133.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:56:38 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 133.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:56:48 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 132.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:56:58 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 131.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:57:08 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 131.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:57:18 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 130.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:57:28 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 130.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:57:38 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 129.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:57:48 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 129.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:57:58 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 128.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:58:08 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 128.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:58:18 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 127.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:58:28 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 127.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:58:38 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 127.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:58:48 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 126.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:58:58 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 126.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:59:08 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 125.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:59:18 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 125.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 7.3% | |
| (MainProcess pid=138633) INFO 10-31 23:59:29 [loggers.py:335] Engine 000: Avg prompt throughput: 15678.4 tokens/s, Avg generation throughput: 45.9 tokens/s, Running: 28 reqs, Waiting: 170 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 6.0% | |
| (MainProcess pid=138633) INFO 10-31 23:59:40 [loggers.py:335] Engine 000: Avg prompt throughput: 26755.6 tokens/s, Avg generation throughput: 14.9 tokens/s, Running: 28 reqs, Waiting: 8 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 5.3% | |
| {"board": "banking77", "closed": 299, "of": 300, "tokens_median": 801.5} | |
| {"saved": "data/jevals/runs/rl60_think.json"} | |
| (MainProcess pid=138633) INFO 10-31 23:59:42 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=138904) INFO 10-31 23:59:42 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=138904) INFO 10-31 23:59:42 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=138904) INFO 10-31 23:59:42 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=138904) INFO 10-31 23:59:42 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=159028) INFO 11-01 04:04:22 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'openbmb/MiniCPM5-2B'} | |
| (MainProcess pid=159028) INFO 11-01 04:04:22 [model.py:733] Resolved architecture: LlamaForCausalLM | |
| (MainProcess pid=159028) INFO 11-01 04:04:22 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=159028) INFO 11-01 04:04:22 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=159028) INFO 11-01 04:04:22 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| Fetching 10 files: 0%| | 0/10 [00:00<?, ?it/s] Fetching 10 files: 100%|██████████| 10/10 [00:00<00:00, 1505.17it/s] | |
| (EngineCore pid=159310) INFO 11-01 04:04:25 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='openbmb/MiniCPM5-2B', speculative_config=None, tokenizer='openbmb/MiniCPM5-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=openbmb/MiniCPM5-2B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=159310) INFO 11-01 04:04:26 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_7d9e4b99dad840ec80d9861e0172533c backend=nccl | |
| (EngineCore pid=159310) INFO 11-01 04:04:26 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=159310) INFO 11-01 04:04:26 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=159310) INFO 11-01 04:04:27 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=159310) INFO 11-01 04:04:29 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=159310) INFO 11-01 04:04:29 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=159310) INFO 11-01 04:04:29 [base_loader.py:82] Initializing model took 1.634409 seconds | |
| (EngineCore pid=159310) INFO 11-01 04:04:29 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 128.92 GiB. | |
| (EngineCore pid=159310) INFO 11-01 04:04:29 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.02it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.01it/s] | |
| (EngineCore pid=159310) INFO 11-01 04:04:30 [default_loader.py:501] Loading weights took 0.50 seconds | |
| (EngineCore pid=159310) INFO 11-01 04:04:32 [model_runner.py:424] Model loading took 4.75 GiB memory and 4.631278 seconds | |
| (EngineCore pid=159310) INFO 11-01 04:04:32 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=159310) INFO 11-01 04:04:32 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=159310) INFO 11-01 04:04:32 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=159310) INFO 11-01 04:04:33 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=3 num_submods=43 | |
| (EngineCore pid=159310) INFO 11-01 04:04:33 [decorators.py:319] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/b5dc5e04701a3cf6fd20d9922eb06fcda1ba1df72a6ff929f3731df693799890/rank_0_0/model | |
| (EngineCore pid=159310) INFO 11-01 04:04:33 [monitor.py:62] [backbone] torch.compile took 0.38 s in total | |
| (EngineCore pid=159310) INFO 11-01 04:04:33 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.19 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:03, 12.34it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:00<00:03, 12.95it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:00<00:03, 12.59it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:00<00:03, 12.95it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:00<00:03, 13.39it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:00<00:02, 13.77it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:01<00:02, 13.89it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:01<00:02, 13.36it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:01<00:02, 12.18it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:01<00:02, 12.66it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:01<00:02, 13.07it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:01<00:02, 13.28it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:01<00:01, 13.43it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:02<00:01, 13.30it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:04<00:09, 2.14it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:05<00:06, 2.87it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:05<00:04, 3.78it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:05<00:03, 4.83it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:05<00:02, 5.97it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:05<00:01, 7.20it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:05<00:01, 8.44it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:05<00:00, 9.60it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:06<00:00, 10.41it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:06<00:00, 10.88it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:06<00:00, 11.56it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:06<00:00, 7.91it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 22.05it/s] | |
| (EngineCore pid=159310) INFO 11-01 04:04:43 [model_runner.py:1127] Graph capturing finished in 7 secs, took 0.42 GiB | |
| (EngineCore pid=159310) INFO 11-01 04:04:43 [gpu_worker.py:759] Available KV cache memory: 65.83 GiB | |
| (EngineCore pid=159310) INFO 11-01 04:04:43 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9140 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9260. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=159310) INFO 11-01 04:04:43 [kv_cache_utils.py:2327] GPU KV cache size: 1,643,616 tokens, Maximum concurrency for 32,768 tokens per request: 50.16x | |
| (EngineCore pid=159310) INFO 11-01 04:04:43 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=159310) INFO 11-01 04:04:43 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:03, 13.03it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:00<00:03, 13.92it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:00<00:03, 13.97it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:00<00:02, 14.45it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:00<00:02, 14.79it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:00<00:02, 15.02it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:00<00:02, 15.10it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:01<00:02, 15.24it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:01<00:02, 14.28it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:01<00:02, 14.29it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:01<00:02, 14.32it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:01<00:01, 14.33it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:01<00:01, 14.33it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:01<00:01, 14.35it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:02<00:01, 14.11it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:02<00:01, 14.12it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:02<00:01, 14.14it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:02<00:01, 14.15it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:02<00:00, 14.17it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:02<00:00, 14.21it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:02<00:00, 14.26it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:03<00:00, 14.35it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:03<00:00, 14.42it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:03<00:00, 14.39it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:03<00:00, 14.02it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 14.31it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 23.88it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 23.91it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 23.83it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:00, 23.70it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 23.11it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 23.43it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 23.92it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 23.83it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 24.17it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 24.67it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 24.63it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 23.53it/s] | |
| (EngineCore pid=159310) INFO 11-01 04:04:49 [model_runner.py:1127] Graph capturing finished in 6 secs, took 0.26 GiB | |
| (EngineCore pid=159310) INFO 11-01 04:04:49 [gpu_worker.py:971] CUDA graph pool memory: 0.26 GiB (actual), 0.47 GiB (estimated), difference: 0.22 GiB (84.1%). | |
| (EngineCore pid=159310) INFO 11-01 04:04:49 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 5.1 GiB for consumed memory (weights + non-torch), 1.97 GiB for peak activation, and 0.26 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=70255142114` (65.43 GiB) to fit into requested memory, or `--kv-cache-memory=76619311616` (71.36 GiB) to fully utilize gpu memory. Current kv cache memory in use is 65.83 GiB. | |
| (EngineCore pid=159310) INFO 11-01 04:04:50 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=159310) INFO 11-01 04:04:51 [core.py:395] init engine (profile, create kv cache, warmup model) took 18.99 s (compilation: 0.38 s) | |
| (EngineCore pid=159310) INFO 11-01 04:04:51 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=159028) INFO 11-01 04:04:52 [hf.py:648] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=159310) WARNING 11-01 04:04:52 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=159028) INFO 11-01 04:05:05 [loggers.py:335] Engine 000: Avg prompt throughput: 38336.7 tokens/s, Avg generation throughput: 80.6 tokens/s, Running: 76 reqs, Waiting: 293 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 12.4% | |
| (MainProcess pid=159028) INFO 11-01 04:05:15 [loggers.py:335] Engine 000: Avg prompt throughput: 51964.0 tokens/s, Avg generation throughput: 126.0 tokens/s, Running: 78 reqs, Waiting: 204 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 19.0% | |
| (MainProcess pid=159028) INFO 11-01 04:05:25 [loggers.py:335] Engine 000: Avg prompt throughput: 51912.1 tokens/s, Avg generation throughput: 127.1 tokens/s, Running: 78 reqs, Waiting: 125 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 18.5% | |
| (MainProcess pid=159028) INFO 11-01 04:05:36 [loggers.py:335] Engine 000: Avg prompt throughput: 49417.8 tokens/s, Avg generation throughput: 121.2 tokens/s, Running: 78 reqs, Waiting: 50 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 17.7% | |
| {"saved": "data/jevals/runs/base_2b.json"} | |
| (MainProcess pid=159028) INFO 11-01 04:05:39 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=159310) INFO 11-01 04:05:39 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=159310) INFO 11-01 04:05:39 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=159310) INFO 11-01 04:05:39 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=159310) INFO 11-01 04:05:39 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=159954) INFO 11-01 04:05:56 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'runs/v2_2b/checkpoints/v2b_joint_merged'} | |
| (MainProcess pid=159954) INFO 11-01 04:05:56 [model.py:733] Resolved architecture: LlamaForCausalLM | |
| (MainProcess pid=159954) INFO 11-01 04:05:56 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=159954) INFO 11-01 04:05:56 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=159954) INFO 11-01 04:05:56 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=160228) INFO 11-01 04:05:58 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='runs/v2_2b/checkpoints/v2b_joint_merged', speculative_config=None, tokenizer='runs/v2_2b/checkpoints/v2b_joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=runs/v2_2b/checkpoints/v2b_joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=160228) INFO 11-01 04:05:59 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_2eecf3fc960449c6a48732a2a892bcf3 backend=nccl | |
| (EngineCore pid=160228) INFO 11-01 04:05:59 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=160228) INFO 11-01 04:05:59 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=160228) INFO 11-01 04:06:01 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=160228) INFO 11-01 04:06:02 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=160228) INFO 11-01 04:06:02 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=160228) INFO 11-01 04:06:02 [base_loader.py:82] Initializing model took 1.424040 seconds | |
| (EngineCore pid=160228) INFO 11-01 04:06:02 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 128.96 GiB. | |
| (EngineCore pid=160228) INFO 11-01 04:06:02 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.62it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.62it/s] | |
| (EngineCore pid=160228) INFO 11-01 04:06:03 [default_loader.py:501] Loading weights took 0.64 seconds | |
| (EngineCore pid=160228) INFO 11-01 04:06:04 [model_runner.py:424] Model loading took 4.75 GiB memory and 3.824139 seconds | |
| (EngineCore pid=160228) INFO 11-01 04:06:04 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=160228) INFO 11-01 04:06:04 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=160228) INFO 11-01 04:06:04 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=160228) INFO 11-01 04:06:05 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=160228) INFO 11-01 04:06:14 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/b08635f8d6/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=160228) INFO 11-01 04:06:14 [backends.py:1166] [backbone] Dynamo bytecode transform time: 9.11 s | |
| (EngineCore pid=160228) INFO 11-01 04:06:23 [backends.py:398] [backbone] Compiling a graph for compile range (1, 32768) takes 8.95 s | |
| (EngineCore pid=160228) INFO 11-01 04:06:29 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4489681 bytes total | |
| (EngineCore pid=160228) INFO 11-01 04:06:29 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/7f7c65dfeef95daecb970119ef8add138199ea9f38b3e66d5929e411f9610ba9/rank_0_0/model | |
| (EngineCore pid=160228) INFO 11-01 04:06:29 [monitor.py:62] [backbone] torch.compile took 23.77 s in total | |
| (EngineCore pid=160228) INFO 11-01 04:06:29 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.17 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:03, 12.55it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:00<00:03, 13.39it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:00<00:03, 13.80it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:00<00:03, 14.17it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:00<00:02, 14.10it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:00<00:02, 14.37it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:00<00:02, 14.52it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:01<00:02, 14.52it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:01<00:02, 13.10it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:01<00:02, 13.44it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:01<00:02, 13.67it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:01<00:01, 13.77it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:01<00:01, 13.26it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:02<00:01, 13.50it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:02<00:01, 13.68it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:02<00:01, 13.80it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:02<00:01, 13.92it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:02<00:01, 13.82it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:02<00:00, 13.90it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:02<00:00, 13.98it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:03<00:00, 14.04it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:03<00:00, 14.10it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:03<00:00, 14.11it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:03<00:00, 13.53it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:03<00:00, 13.62it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 13.73it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 22.77it/s] | |
| (EngineCore pid=160228) INFO 11-01 04:06:36 [model_runner.py:1127] Graph capturing finished in 5 secs, took 0.42 GiB | |
| (EngineCore pid=160228) INFO 11-01 04:06:37 [gpu_worker.py:759] Available KV cache memory: 65.84 GiB | |
| (EngineCore pid=160228) INFO 11-01 04:06:37 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9140 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9260. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=160228) INFO 11-01 04:06:37 [kv_cache_utils.py:2327] GPU KV cache size: 1,643,712 tokens, Maximum concurrency for 32,768 tokens per request: 50.16x | |
| (EngineCore pid=160228) INFO 11-01 04:06:37 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=160228) INFO 11-01 04:06:37 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:03, 12.75it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:00<00:03, 13.38it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:00<00:03, 13.68it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:00<00:03, 13.84it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:00<00:02, 14.25it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:00<00:02, 14.53it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:00<00:02, 14.84it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:01<00:02, 15.06it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:03<00:16, 2.06it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:04<00:11, 2.80it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:04<00:07, 3.71it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:04<00:05, 4.78it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:04<00:04, 5.95it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:04<00:03, 7.20it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:04<00:02, 8.43it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:04<00:01, 9.54it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:05<00:01, 10.57it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:05<00:01, 11.41it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:05<00:01, 12.09it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:05<00:00, 12.62it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:05<00:00, 13.05it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:05<00:00, 13.37it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:05<00:00, 13.59it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:06<00:00, 13.74it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:06<00:00, 13.84it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:06<00:00, 8.13it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 23.41it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 23.51it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 23.48it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:00, 23.64it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 23.56it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 23.86it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 24.09it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 23.84it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 24.04it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 24.40it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 24.40it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 24.06it/s] | |
| (EngineCore pid=160228) INFO 11-01 04:07:03 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.26 GiB | |
| (EngineCore pid=160228) INFO 11-01 04:07:03 [gpu_worker.py:971] CUDA graph pool memory: 0.26 GiB (actual), 0.47 GiB (estimated), difference: 0.22 GiB (84.1%). | |
| (EngineCore pid=160228) INFO 11-01 04:07:03 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 5.1 GiB for consumed memory (weights + non-torch), 1.97 GiB for peak activation, and 0.26 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=70259336418` (65.43 GiB) to fit into requested memory, or `--kv-cache-memory=76623505920` (71.36 GiB) to fully utilize gpu memory. Current kv cache memory in use is 65.84 GiB. | |
| (EngineCore pid=160228) INFO 11-01 04:07:03 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=160228) INFO 11-01 04:07:04 [core.py:395] init engine (profile, create kv cache, warmup model) took 59.43 s (compilation: 23.77 s) | |
| (EngineCore pid=160228) INFO 11-01 04:07:04 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=159954) INFO 11-01 04:07:05 [hf.py:648] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=160228) WARNING 11-01 04:07:05 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=159954) INFO 11-01 04:07:16 [loggers.py:335] Engine 000: Avg prompt throughput: 44612.2 tokens/s, Avg generation throughput: 94.0 tokens/s, Running: 76 reqs, Waiting: 265 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 12.4% | |
| (MainProcess pid=159954) INFO 11-01 04:07:26 [loggers.py:335] Engine 000: Avg prompt throughput: 52069.8 tokens/s, Avg generation throughput: 124.5 tokens/s, Running: 78 reqs, Waiting: 203 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 17.3% | |
| (MainProcess pid=159954) INFO 11-01 04:07:36 [loggers.py:335] Engine 000: Avg prompt throughput: 51976.9 tokens/s, Avg generation throughput: 125.6 tokens/s, Running: 79 reqs, Waiting: 124 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 17.3% | |
| (MainProcess pid=159954) INFO 11-01 04:07:47 [loggers.py:335] Engine 000: Avg prompt throughput: 51946.9 tokens/s, Avg generation throughput: 125.8 tokens/s, Running: 78 reqs, Waiting: 42 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 17.2% | |
| {"saved": "data/jevals/runs/v2_2b.json"} | |
| (MainProcess pid=159954) INFO 11-01 04:07:50 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=160228) INFO 11-01 04:07:50 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=160228) INFO 11-01 04:07:50 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=160228) INFO 11-01 04:07:50 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=160228) INFO 11-01 04:07:50 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=231216) INFO 11-01 05:28:51 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'runs/v2_2b/checkpoints/v2b_ro_merged'} | |
| (MainProcess pid=231216) INFO 11-01 05:28:51 [model.py:733] Resolved architecture: LlamaForCausalLM | |
| (MainProcess pid=231216) INFO 11-01 05:28:51 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=231216) INFO 11-01 05:28:51 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=231216) INFO 11-01 05:28:51 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=231483) INFO 11-01 05:28:53 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='runs/v2_2b/checkpoints/v2b_ro_merged', speculative_config=None, tokenizer='runs/v2_2b/checkpoints/v2b_ro_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=runs/v2_2b/checkpoints/v2b_ro_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=231483) INFO 11-01 05:28:54 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_f1d1ce9e301f42e68a44b595dde24272 backend=nccl | |
| (EngineCore pid=231483) INFO 11-01 05:28:54 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=231483) INFO 11-01 05:28:54 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=231483) INFO 11-01 05:28:56 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=231483) INFO 11-01 05:28:57 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=231483) INFO 11-01 05:28:57 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=231483) INFO 11-01 05:28:57 [base_loader.py:82] Initializing model took 1.425358 seconds | |
| (EngineCore pid=231483) INFO 11-01 05:28:57 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 122.69 GiB. | |
| (EngineCore pid=231483) INFO 11-01 05:28:57 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.92it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.92it/s] | |
| (EngineCore pid=231483) INFO 11-01 05:28:58 [default_loader.py:501] Loading weights took 0.54 seconds | |
| (EngineCore pid=231483) INFO 11-01 05:28:59 [model_runner.py:424] Model loading took 4.75 GiB memory and 3.744601 seconds | |
| (EngineCore pid=231483) INFO 11-01 05:28:59 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=231483) INFO 11-01 05:28:59 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=231483) INFO 11-01 05:28:59 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=231483) INFO 11-01 05:29:00 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=231483) INFO 11-01 05:29:09 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/1ffbc07d3a/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=231483) INFO 11-01 05:29:09 [backends.py:1166] [backbone] Dynamo bytecode transform time: 9.20 s | |
| (EngineCore pid=231483) INFO 11-01 05:29:16 [backends.py:398] [backbone] Compiling a graph for compile range (1, 32768) takes 6.57 s | |
| (EngineCore pid=231483) INFO 11-01 05:29:22 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4489635 bytes total | |
| (EngineCore pid=231483) INFO 11-01 05:29:22 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/049dfac80b7c5f5c6e13b2fb8eb1f5dd104ef7259d11ace5e754e1e8e8447679/rank_0_0/model | |
| (EngineCore pid=231483) INFO 11-01 05:29:22 [monitor.py:62] [backbone] torch.compile took 21.51 s in total | |
| (EngineCore pid=231483) INFO 11-01 05:29:22 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.17 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:03, 12.58it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:00<00:03, 13.19it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:00<00:03, 13.62it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:00<00:03, 13.87it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:00<00:02, 14.25it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:00<00:02, 14.45it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:00<00:02, 14.63it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:01<00:02, 14.39it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:01<00:02, 12.91it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:01<00:02, 13.22it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:01<00:02, 13.44it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:01<00:02, 13.49it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:01<00:01, 13.57it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:02<00:01, 13.66it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:02<00:01, 13.77it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:02<00:01, 13.86it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:02<00:01, 13.99it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:02<00:01, 13.96it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:02<00:00, 14.04it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:02<00:00, 14.08it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:03<00:00, 14.14it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:03<00:00, 14.19it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:03<00:00, 14.08it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:03<00:00, 13.54it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:03<00:00, 13.63it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 13.74it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 22.74it/s] | |
| (EngineCore pid=231483) INFO 11-01 05:29:29 [model_runner.py:1127] Graph capturing finished in 5 secs, took 0.42 GiB | |
| (EngineCore pid=231483) INFO 11-01 05:29:29 [gpu_worker.py:759] Available KV cache memory: 65.84 GiB | |
| (EngineCore pid=231483) INFO 11-01 05:29:29 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9140 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9260. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=231483) INFO 11-01 05:29:29 [kv_cache_utils.py:2327] GPU KV cache size: 1,643,712 tokens, Maximum concurrency for 32,768 tokens per request: 50.16x | |
| (EngineCore pid=231483) INFO 11-01 05:29:30 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=231483) INFO 11-01 05:29:30 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:03, 12.75it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:00<00:03, 13.51it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:00<00:03, 13.86it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:00<00:03, 14.18it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:00<00:02, 14.57it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:00<00:02, 14.88it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:00<00:02, 15.15it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:01<00:02, 15.38it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:01<00:02, 14.46it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:01<00:02, 14.51it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:01<00:01, 14.53it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:01<00:01, 14.49it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:01<00:01, 14.45it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:01<00:01, 14.37it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:02<00:01, 14.33it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:02<00:01, 14.26it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:02<00:01, 14.26it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:02<00:01, 14.25it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:02<00:00, 14.28it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:02<00:00, 14.27it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:02<00:00, 14.27it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:03<00:00, 14.29it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:03<00:00, 14.35it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:03<00:00, 14.34it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:03<00:00, 14.28it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 14.38it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 24.19it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 24.13it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 24.06it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:00, 23.47it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 23.59it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 24.03it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 24.35it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 24.08it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 24.32it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 24.69it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 24.78it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 24.36it/s] | |
| (EngineCore pid=231483) INFO 11-01 05:29:53 [model_runner.py:1127] Graph capturing finished in 6 secs, took 0.26 GiB | |
| (EngineCore pid=231483) INFO 11-01 05:29:53 [gpu_worker.py:971] CUDA graph pool memory: 0.26 GiB (actual), 0.47 GiB (estimated), difference: 0.22 GiB (84.1%). | |
| (EngineCore pid=231483) INFO 11-01 05:29:53 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 5.1 GiB for consumed memory (weights + non-torch), 1.97 GiB for peak activation, and 0.26 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=70259336418` (65.43 GiB) to fit into requested memory, or `--kv-cache-memory=76623505920` (71.36 GiB) to fully utilize gpu memory. Current kv cache memory in use is 65.84 GiB. | |
| (EngineCore pid=231483) INFO 11-01 05:29:54 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=231483) INFO 11-01 05:29:54 [core.py:395] init engine (profile, create kv cache, warmup model) took 54.84 s (compilation: 21.51 s) | |
| (EngineCore pid=231483) INFO 11-01 05:29:54 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=231216) INFO 11-01 05:29:55 [hf.py:648] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=231483) WARNING 11-01 05:29:56 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=231216) INFO 11-01 05:30:06 [loggers.py:335] Engine 000: Avg prompt throughput: 44732.4 tokens/s, Avg generation throughput: 94.1 tokens/s, Running: 76 reqs, Waiting: 285 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 12.4% | |
| (MainProcess pid=231216) INFO 11-01 05:30:17 [loggers.py:335] Engine 000: Avg prompt throughput: 51989.6 tokens/s, Avg generation throughput: 124.3 tokens/s, Running: 78 reqs, Waiting: 203 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 17.5% | |
| (MainProcess pid=231216) INFO 11-01 05:30:27 [loggers.py:335] Engine 000: Avg prompt throughput: 51949.9 tokens/s, Avg generation throughput: 125.5 tokens/s, Running: 79 reqs, Waiting: 119 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 17.5% | |
| (MainProcess pid=231216) INFO 11-01 05:30:37 [loggers.py:335] Engine 000: Avg prompt throughput: 51861.3 tokens/s, Avg generation throughput: 125.5 tokens/s, Running: 79 reqs, Waiting: 28 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 16.9% | |
| {"saved": "data/jevals/runs/v2_2b_ro.json"} | |
| (MainProcess pid=231216) INFO 11-01 05:30:40 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=231483) INFO 11-01 05:30:40 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=231483) INFO 11-01 05:30:40 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=231483) INFO 11-01 05:30:40 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=231483) INFO 11-01 05:30:40 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=231216) WARNING 11-01 05:30:45 [utils.py:648] [shutdown] Process manager: force killing remaining processes count=1 | |
| (MainProcess pid=231216) WARNING 11-01 05:30:45 [utils.py:653] [shutdown] Process manager: force killing remaining process EngineCore pid 231483 | |
| (MainProcess pid=232208) INFO 11-01 05:30:59 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 384, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'runs/v2/checkpoints/v2_joint_merged'} | |
| (MainProcess pid=232208) INFO 11-01 05:30:59 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM | |
| (MainProcess pid=232208) INFO 11-01 05:30:59 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=232208) INFO 11-01 05:30:59 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=232208) INFO 11-01 05:30:59 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled | |
| (MainProcess pid=232208) INFO 11-01 05:30:59 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=232482) INFO 11-01 05:31:08 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='runs/v2/checkpoints/v2_joint_merged', speculative_config=None, tokenizer='runs/v2/checkpoints/v2_joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=runs/v2/checkpoints/v2_joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=232482) INFO 11-01 05:31:09 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_158940e539004071883f6f3931d1f0b0 backend=nccl | |
| (EngineCore pid=232482) INFO 11-01 05:31:09 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=232482) INFO 11-01 05:31:09 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=232482) INFO 11-01 05:31:11 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=232482) INFO 11-01 05:31:11 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=232482) INFO 11-01 05:31:12 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=232482) INFO 11-01 05:31:12 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=232482) INFO 11-01 05:31:12 [base_loader.py:82] Initializing model took 1.550776 seconds | |
| (EngineCore pid=232482) INFO 11-01 05:31:12 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 122.13 GiB. | |
| (EngineCore pid=232482) INFO 11-01 05:31:12 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.15it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.15it/s] | |
| (EngineCore pid=232482) INFO 11-01 05:31:13 [default_loader.py:501] Loading weights took 0.90 seconds | |
| (EngineCore pid=232482) INFO 11-01 05:31:15 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.371020 seconds | |
| (EngineCore pid=232482) INFO 11-01 05:31:15 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=232482) INFO 11-01 05:31:15 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=232482) INFO 11-01 05:31:15 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=232482) INFO 11-01 05:31:15 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=232482) INFO 11-01 05:31:15 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=232482) INFO 11-01 05:31:16 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=232482) INFO 11-01 05:31:21 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/0367f23ba9/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=232482) INFO 11-01 05:31:21 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.41 s | |
| (EngineCore pid=232482) INFO 11-01 05:31:37 [backends.py:398] [backbone] Compiling a graph for compile range (1, 32768) takes 16.21 s | |
| (EngineCore pid=232482) INFO 11-01 05:31:40 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20354827 bytes total | |
| (EngineCore pid=232482) INFO 11-01 05:31:40 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/09467e65e3308d125f9d9eb27cd1c779fe5878dc64eb73b45e25197753a6a578/rank_0_0/model | |
| (EngineCore pid=232482) INFO 11-01 05:31:40 [monitor.py:62] [backbone] torch.compile took 24.68 s in total | |
| (EngineCore pid=232482) INFO 11-01 05:33:08 [monitor.py:91] [backbone] Initial profiling/warmup run took 87.83 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:01<01:03, 1.27s/it] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:33, 1.47it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:01<00:23, 2.05it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:02<00:17, 2.64it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:02<00:15, 2.92it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:02<00:13, 3.23it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:02<00:13, 3.31it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:03<00:11, 3.77it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:21, 1.95it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:04<00:11, 3.39it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:04<00:06, 5.83it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:04<00:04, 8.35it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:04<00:04, 7.91it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:04<00:02, 10.33it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:05<00:02, 12.59it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:05<00:01, 14.52it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:05<00:01, 16.11it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:05<00:00, 17.26it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:05<00:00, 18.08it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:05<00:00, 18.86it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:05<00:00, 19.58it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:06<00:00, 20.15it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:06<00:00, 20.49it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:06<00:00, 7.94it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 23.66it/s] | |
| (EngineCore pid=232482) INFO 11-01 05:33:20 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.42 GiB | |
| (EngineCore pid=232482) INFO 11-01 05:33:21 [gpu_worker.py:759] Available KV cache memory: 58.78 GiB | |
| (EngineCore pid=232482) INFO 11-01 05:33:21 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9138 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9262. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=232482) INFO 11-01 05:33:21 [kv_cache_utils.py:2327] GPU KV cache size: 1,731,955 tokens, Maximum concurrency for 32,768 tokens per request: 52.86x | |
| (EngineCore pid=232482) INFO 11-01 05:33:21 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=232482) INFO 11-01 05:33:21 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=232482) INFO 11-01 05:33:21 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=232482) INFO 11-01 05:33:21 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=232482) INFO 11-01 05:33:21 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=232482) INFO 11-01 05:33:22 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:16, 3.02it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:17, 2.78it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:01<00:16, 2.84it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:01<00:14, 3.16it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:01<00:13, 3.32it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:01<00:12, 3.56it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:02<00:11, 3.95it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:02<00:09, 4.64it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:02<00:04, 8.42it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:02<00:03, 11.63it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:02<00:02, 14.28it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:02<00:01, 16.41it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:02<00:01, 18.08it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:02<00:01, 19.21it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:03<00:01, 20.08it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:03<00:00, 20.63it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:03<00:00, 20.85it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:03<00:00, 21.23it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:03<00:00, 21.48it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:03<00:00, 21.66it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:03<00:00, 21.89it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:04<00:00, 22.11it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:04<00:00, 12.40it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/43 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 7%|▋ | 3/43 [00:00<00:01, 26.72it/s] Capturing CUDA graphs (FULL): 14%|█▍ | 6/43 [00:00<00:01, 25.83it/s] Capturing CUDA graphs (FULL): 21%|██ | 9/43 [00:00<00:01, 26.29it/s] Capturing CUDA graphs (FULL): 28%|██▊ | 12/43 [00:00<00:01, 26.83it/s] Capturing CUDA graphs (FULL): 35%|███▍ | 15/43 [00:00<00:01, 27.25it/s] Capturing CUDA graphs (FULL): 42%|████▏ | 18/43 [00:00<00:00, 27.39it/s] Capturing CUDA graphs (FULL): 49%|████▉ | 21/43 [00:00<00:00, 27.50it/s] Capturing CUDA graphs (FULL): 56%|█████▌ | 24/43 [00:00<00:00, 27.29it/s] Capturing CUDA graphs (FULL): 63%|██████▎ | 27/43 [00:00<00:00, 27.31it/s] Capturing CUDA graphs (FULL): 70%|██████▉ | 30/43 [00:01<00:00, 27.45it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 33/43 [00:01<00:00, 27.65it/s] Capturing CUDA graphs (FULL): 84%|████████▎ | 36/43 [00:01<00:00, 27.90it/s] Capturing CUDA graphs (FULL): 91%|█████████ | 39/43 [00:01<00:00, 27.83it/s] Capturing CUDA graphs (FULL): 98%|█████████▊| 42/43 [00:01<00:00, 28.04it/s] Capturing CUDA graphs (FULL): 100%|██████████| 43/43 [00:01<00:00, 27.48it/s] | |
| (EngineCore pid=232482) INFO 11-01 05:33:47 [model_runner.py:1127] Graph capturing finished in 7 secs, took 0.29 GiB | |
| (EngineCore pid=232482) INFO 11-01 05:33:47 [gpu_worker.py:971] CUDA graph pool memory: 0.29 GiB (actual), 0.49 GiB (estimated), difference: 0.2 GiB (66.7%). | |
| (EngineCore pid=232482) INFO 11-01 05:33:47 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 10.65 GiB for consumed memory (weights + non-torch), 3.49 GiB for peak activation, and 0.29 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=62638283490` (58.34 GiB) to fit into requested memory, or `--kv-cache-memory=69002452992` (64.26 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.78 GiB. | |
| (EngineCore pid=232482) INFO 11-01 05:33:48 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=232482) INFO 11-01 05:33:49 [core.py:395] init engine (profile, create kv cache, warmup model) took 153.85 s (compilation: 24.68 s) | |
| (EngineCore pid=232482) INFO 11-01 05:33:49 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=232208) INFO 11-01 05:33:50 [hf.py:648] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| (MainProcess pid=232208) INFO 11-01 05:34:01 [loggers.py:335] Engine 000: Avg prompt throughput: 11420.5 tokens/s, Avg generation throughput: 3173.2 tokens/s, Running: 300 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.2%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:34:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7257.4 tokens/s, Running: 295 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.6%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:34:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6896.3 tokens/s, Running: 214 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.5%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:34:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6002.9 tokens/s, Running: 123 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:34:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5185.8 tokens/s, Running: 79 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.3%, Prefix cache hit rate: 0.0% | |
| (EngineCore pid=232482) WARNING 11-01 05:34:46 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=232482) WARNING 11-01 05:34:46 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=232482) WARNING 11-01 05:34:47 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=232208) INFO 11-01 05:34:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3353.3 tokens/s, Running: 55 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.2%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:35:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2749.9 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:35:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2153.7 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:35:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1915.8 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:35:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 991.7 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:35:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 706.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:35:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 388.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:36:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 253.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:36:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 251.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:36:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 249.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:36:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 248.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:36:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 246.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:36:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 245.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:37:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 243.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:37:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 242.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:37:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 240.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:37:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 239.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:37:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 238.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:37:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 237.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:38:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 235.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:38:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 234.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:38:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 233.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:38:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 232.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:38:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 231.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:38:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 229.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:39:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 228.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:39:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 227.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=232208) INFO 11-01 05:39:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 226.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0% | |
| (EngineCore pid=232482) WARNING 11-01 05:39:24 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=232208) INFO 11-01 05:39:31 [loggers.py:335] Engine 000: Avg prompt throughput: 16282.0 tokens/s, Avg generation throughput: 63.6 tokens/s, Running: 41 reqs, Waiting: 163 reqs, GPU KV cache usage: 9.8%, Prefix cache hit rate: 4.2% | |
| (MainProcess pid=232208) INFO 11-01 05:39:42 [loggers.py:335] Engine 000: Avg prompt throughput: 26967.4 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 5.3% | |
| {"board": "pubmedqa", "closed": 298, "of": 300, "tokens_median": 885.0} | |
| (MainProcess pid=232208) INFO 11-01 05:39:52 [loggers.py:335] Engine 000: Avg prompt throughput: 25212.1 tokens/s, Avg generation throughput: 1003.8 tokens/s, Running: 300 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:40:02 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7088.8 tokens/s, Running: 300 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:40:12 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6823.5 tokens/s, Running: 284 reqs, Waiting: 0 reqs, GPU KV cache usage: 45.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:40:22 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6548.4 tokens/s, Running: 228 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:40:32 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6189.0 tokens/s, Running: 200 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:40:42 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5815.1 tokens/s, Running: 187 reqs, Waiting: 0 reqs, GPU KV cache usage: 38.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:40:52 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5559.5 tokens/s, Running: 181 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:41:02 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5324.3 tokens/s, Running: 174 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:41:12 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5104.0 tokens/s, Running: 170 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:41:22 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4903.4 tokens/s, Running: 163 reqs, Waiting: 0 reqs, GPU KV cache usage: 43.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:41:32 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4711.9 tokens/s, Running: 159 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:41:42 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4556.4 tokens/s, Running: 152 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:41:52 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4304.3 tokens/s, Running: 143 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:42:02 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4094.9 tokens/s, Running: 133 reqs, Waiting: 0 reqs, GPU KV cache usage: 43.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:42:12 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4170.6 tokens/s, Running: 118 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:42:22 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3973.9 tokens/s, Running: 106 reqs, Waiting: 0 reqs, GPU KV cache usage: 38.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:42:32 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3894.2 tokens/s, Running: 100 reqs, Waiting: 0 reqs, GPU KV cache usage: 38.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:42:42 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3596.2 tokens/s, Running: 88 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:42:52 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3309.5 tokens/s, Running: 83 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:43:02 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3238.5 tokens/s, Running: 79 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:43:12 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3220.8 tokens/s, Running: 78 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:43:22 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3062.4 tokens/s, Running: 71 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:43:32 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2862.4 tokens/s, Running: 66 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:43:42 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2709.3 tokens/s, Running: 66 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:43:52 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2634.5 tokens/s, Running: 65 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:44:02 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2572.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:44:12 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2497.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:44:22 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2428.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:44:32 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2379.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:44:42 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2334.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:44:52 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2292.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:45:02 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2251.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:45:12 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2213.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 43.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:45:22 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2176.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 45.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:45:32 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2140.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 46.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:45:42 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2107.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 47.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:45:52 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2074.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 48.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:46:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2043.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 49.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:46:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2013.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:46:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1984.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 51.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:46:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1957.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:46:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1931.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:46:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1905.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 54.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:47:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1881.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 55.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:47:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1857.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 56.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:47:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1835.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 57.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:47:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1813.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 58.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:47:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1792.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 59.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:47:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1772.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 60.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:48:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1752.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 61.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:48:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1733.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 62.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:48:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1716.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 62.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:48:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1698.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 63.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:48:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1681.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:48:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1664.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 65.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:49:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1648.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 66.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:49:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1633.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:49:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1618.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:49:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1603.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:49:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1589.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:49:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1575.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:50:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1562.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:50:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1548.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:50:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1536.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:50:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1523.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:50:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1511.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:50:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1499.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:51:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1487.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:51:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1476.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:51:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1465.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:51:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1454.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:51:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1443.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:51:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1432.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:52:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1422.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:52:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1412.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:52:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1401.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:52:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1392.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:52:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1383.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 83.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:52:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1373.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:53:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1364.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:53:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1355.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:53:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1346.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:53:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1338.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:53:43 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1329.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:53:53 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1321.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:54:03 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1313.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:54:13 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1305.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:54:23 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1297.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:54:33 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1289.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:54:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1282.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:54:54 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1274.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:55:04 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1267.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:55:14 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1260.1 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 94.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:55:24 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1253.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 94.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:55:34 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1246.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 95.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:55:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1239.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 95.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:55:54 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1233.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:56:04 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1226.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:56:14 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1219.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:56:24 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1213.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:56:34 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1207.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 99.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:56:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1201.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:56:55 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 862.7 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:57:05 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 996.4 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 98.8%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:57:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1170.2 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:57:25 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 820.6 tokens/s, Running: 60 reqs, Waiting: 2 reqs, GPU KV cache usage: 98.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:57:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 795.3 tokens/s, Running: 60 reqs, Waiting: 2 reqs, GPU KV cache usage: 98.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:57:46 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1139.4 tokens/s, Running: 60 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:57:56 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 962.1 tokens/s, Running: 59 reqs, Waiting: 3 reqs, GPU KV cache usage: 98.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:58:06 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 574.2 tokens/s, Running: 59 reqs, Waiting: 3 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:58:16 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1109.8 tokens/s, Running: 59 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:58:26 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1104.7 tokens/s, Running: 59 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:58:37 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 764.5 tokens/s, Running: 58 reqs, Waiting: 4 reqs, GPU KV cache usage: 98.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:58:47 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 908.0 tokens/s, Running: 58 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:58:57 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1075.0 tokens/s, Running: 58 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:59:07 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1070.0 tokens/s, Running: 58 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:59:17 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 718.8 tokens/s, Running: 57 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:59:27 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1045.8 tokens/s, Running: 57 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:59:37 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1041.3 tokens/s, Running: 57 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:59:47 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 852.5 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 98.5%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 05:59:57 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 808.9 tokens/s, Running: 56 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.7%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 06:00:07 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1017.0 tokens/s, Running: 56 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.3%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 06:00:17 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 177.5 tokens/s, Running: 56 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 06:00:28 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 885.0 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.0%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 06:00:38 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1163.7 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.2%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 06:00:48 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 950.7 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.4%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 06:00:58 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 419.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 06:01:08 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 266.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 4.3% | |
| (MainProcess pid=232208) INFO 11-01 06:01:19 [loggers.py:335] Engine 000: Avg prompt throughput: 19270.1 tokens/s, Avg generation throughput: 24.5 tokens/s, Running: 20 reqs, Waiting: 154 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 3.2% | |
| (MainProcess pid=232208) INFO 11-01 06:01:30 [loggers.py:335] Engine 000: Avg prompt throughput: 26680.5 tokens/s, Avg generation throughput: 9.2 tokens/s, Running: 21 reqs, Waiting: 51 reqs, GPU KV cache usage: 6.1%, Prefix cache hit rate: 2.7% | |
| {"board": "helpsteer2", "closed": 238, "of": 300, "tokens_median": 3181.0} | |
| (MainProcess pid=232208) INFO 11-01 06:01:41 [loggers.py:335] Engine 000: Avg prompt throughput: 28453.3 tokens/s, Avg generation throughput: 25.4 tokens/s, Running: 211 reqs, Waiting: 89 reqs, GPU KV cache usage: 23.1%, Prefix cache hit rate: 2.0% | |
| (MainProcess pid=232208) INFO 11-01 06:01:51 [loggers.py:335] Engine 000: Avg prompt throughput: 7222.5 tokens/s, Avg generation throughput: 5443.4 tokens/s, Running: 300 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.1%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:02:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7182.0 tokens/s, Running: 287 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.4%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:02:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6777.0 tokens/s, Running: 186 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.6%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:02:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5730.0 tokens/s, Running: 111 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.0%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:02:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5190.4 tokens/s, Running: 91 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.2%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:02:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4564.4 tokens/s, Running: 83 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:02:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4188.2 tokens/s, Running: 70 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:03:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3747.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.0%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:03:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3352.0 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:03:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2804.6 tokens/s, Running: 33 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.8%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:03:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2137.0 tokens/s, Running: 23 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:03:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1787.0 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.3%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:03:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1532.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:04:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1441.3 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:04:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1412.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.2%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:04:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1380.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:04:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1350.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:04:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1322.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.4%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:04:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1295.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.3%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:05:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1271.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.7%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:05:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1248.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.7%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:05:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1225.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.0%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:05:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1204.8 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.5%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:05:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1184.7 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:05:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1165.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:06:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1147.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:06:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1130.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:06:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1114.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:06:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1098.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:06:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1083.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.9%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:06:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1068.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:07:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1054.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:07:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1040.4 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:07:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1027.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.1%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:07:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1015.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:07:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1002.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:07:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 991.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.4%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:08:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 979.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:08:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 969.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:08:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 958.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.1%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:08:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 948.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:08:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 938.7 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.0%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:08:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 929.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:09:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 920.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.9%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:09:11 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 910.7 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:09:21 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 902.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:09:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 893.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:09:41 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 885.9 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.6%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:09:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 878.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:10:01 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 869.9 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.2%, Prefix cache hit rate: 1.5% | |
| (MainProcess pid=232208) INFO 11-01 06:10:12 [loggers.py:335] Engine 000: Avg prompt throughput: 20971.9 tokens/s, Avg generation throughput: 51.7 tokens/s, Running: 40 reqs, Waiting: 128 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.2% | |
| (MainProcess pid=232208) INFO 11-01 06:10:23 [loggers.py:335] Engine 000: Avg prompt throughput: 26903.4 tokens/s, Avg generation throughput: 14.0 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.1% | |
| {"board": "banking77", "closed": 284, "of": 300, "tokens_median": 827.0} | |
| {"saved": "data/jevals/runs/v2_joint_think.json"} | |
| (MainProcess pid=232208) INFO 11-01 06:10:23 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=232482) INFO 11-01 06:10:23 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=232482) INFO 11-01 06:10:23 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=232482) INFO 11-01 06:10:23 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=232482) INFO 11-01 06:10:23 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=232208) WARNING 11-01 06:10:28 [utils.py:648] [shutdown] Process manager: force killing remaining processes count=1 | |
| (MainProcess pid=232208) WARNING 11-01 06:10:28 [utils.py:653] [shutdown] Process manager: force killing remaining process EngineCore pid 232482 | |
| FAILED base_think | |
| FAILED v1_joint_think | |
| (MainProcess pid=235115) INFO 11-01 06:10:50 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 96, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'openbmb/MiniCPM5-2B'} | |
| (MainProcess pid=235115) INFO 11-01 06:10:50 [model.py:733] Resolved architecture: LlamaForCausalLM | |
| (MainProcess pid=235115) INFO 11-01 06:10:50 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=235115) INFO 11-01 06:10:50 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=235115) INFO 11-01 06:10:50 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| Fetching 10 files: 0%| | 0/10 [00:00<?, ?it/s] Fetching 10 files: 100%|██████████| 10/10 [00:00<00:00, 1079.42it/s] | |
| (EngineCore pid=235400) INFO 11-01 06:10:53 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='openbmb/MiniCPM5-2B', speculative_config=None, tokenizer='openbmb/MiniCPM5-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=openbmb/MiniCPM5-2B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 192, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=235400) INFO 11-01 06:10:55 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_8ea3e6b5baf3418396d2a9e91109a40b backend=nccl | |
| (EngineCore pid=235400) INFO 11-01 06:10:55 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=235400) INFO 11-01 06:10:55 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=235400) INFO 11-01 06:10:56 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=235400) INFO 11-01 06:10:57 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=235400) INFO 11-01 06:10:57 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=235400) INFO 11-01 06:10:57 [base_loader.py:82] Initializing model took 1.586055 seconds | |
| (EngineCore pid=235400) INFO 11-01 06:10:58 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 122.48 GiB. | |
| (EngineCore pid=235400) INFO 11-01 06:10:58 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.96it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.96it/s] | |
| (EngineCore pid=235400) INFO 11-01 06:10:58 [default_loader.py:501] Loading weights took 0.52 seconds | |
| (EngineCore pid=235400) INFO 11-01 06:11:00 [model_runner.py:424] Model loading took 4.75 GiB memory and 4.516223 seconds | |
| (EngineCore pid=235400) INFO 11-01 06:11:00 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=235400) INFO 11-01 06:11:00 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=235400) INFO 11-01 06:11:00 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=235400) INFO 11-01 06:11:01 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=235400) INFO 11-01 06:11:11 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/06b11cd84a/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=235400) INFO 11-01 06:11:11 [backends.py:1166] [backbone] Dynamo bytecode transform time: 9.79 s | |
| (EngineCore pid=235400) INFO 11-01 06:11:18 [backends.py:398] [backbone] Compiling a graph for compile range (1, 32768) takes 6.53 s | |
| (EngineCore pid=235400) INFO 11-01 06:11:23 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4489231 bytes total | |
| (EngineCore pid=235400) INFO 11-01 06:11:23 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/7fc6033238bff6fcd94c7e664c76229824fdca25db76978a1a82f8894eb32551/rank_0_0/model | |
| (EngineCore pid=235400) INFO 11-01 06:11:23 [monitor.py:62] [backbone] torch.compile took 22.27 s in total | |
| (EngineCore pid=235400) INFO 11-01 06:11:24 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.17 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/27 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 7%|▋ | 2/27 [00:00<00:01, 17.45it/s] Capturing CUDA graphs (PIECEWISE): 19%|█▊ | 5/27 [00:00<00:01, 20.66it/s] Capturing CUDA graphs (PIECEWISE): 30%|██▉ | 8/27 [00:00<00:00, 21.40it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 11/27 [00:00<00:00, 22.02it/s] Capturing CUDA graphs (PIECEWISE): 52%|█████▏ | 14/27 [00:00<00:00, 21.41it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 17/27 [00:00<00:00, 21.17it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 20/27 [00:00<00:00, 20.84it/s] Capturing CUDA graphs (PIECEWISE): 85%|████████▌ | 23/27 [00:01<00:00, 20.82it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▋| 26/27 [00:01<00:00, 19.74it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:01<00:00, 20.26it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 25.63it/s] | |
| (EngineCore pid=235400) INFO 11-01 06:11:28 [model_runner.py:1127] Graph capturing finished in 2 secs, took 0.25 GiB | |
| (EngineCore pid=235400) INFO 11-01 06:11:29 [gpu_worker.py:759] Available KV cache memory: 66.11 GiB | |
| (EngineCore pid=235400) INFO 11-01 06:11:29 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9163 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9237. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=235400) INFO 11-01 06:11:29 [kv_cache_utils.py:2327] GPU KV cache size: 1,650,544 tokens, Maximum concurrency for 32,768 tokens per request: 50.37x | |
| (EngineCore pid=235400) INFO 11-01 06:11:29 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=235400) INFO 11-01 06:11:29 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/27 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 11%|█ | 3/27 [00:00<00:01, 21.59it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 6/27 [00:00<00:00, 22.48it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 9/27 [00:00<00:00, 22.83it/s] Capturing CUDA graphs (PIECEWISE): 44%|████▍ | 12/27 [00:00<00:00, 23.03it/s] Capturing CUDA graphs (PIECEWISE): 56%|█████▌ | 15/27 [00:01<00:01, 8.50it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 17/27 [00:03<00:03, 2.98it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 20/27 [00:03<00:01, 4.28it/s] Capturing CUDA graphs (PIECEWISE): 85%|████████▌ | 23/27 [00:03<00:00, 5.84it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▋| 26/27 [00:03<00:00, 7.62it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:03<00:00, 7.32it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/15 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 20%|██ | 3/15 [00:00<00:00, 24.26it/s] Capturing CUDA graphs (FULL): 40%|████ | 6/15 [00:00<00:00, 24.69it/s] Capturing CUDA graphs (FULL): 60%|██████ | 9/15 [00:00<00:00, 25.13it/s] Capturing CUDA graphs (FULL): 80%|████████ | 12/15 [00:00<00:00, 25.23it/s] Capturing CUDA graphs (FULL): 100%|██████████| 15/15 [00:00<00:00, 25.39it/s] Capturing CUDA graphs (FULL): 100%|██████████| 15/15 [00:00<00:00, 25.18it/s] | |
| (EngineCore pid=235400) INFO 11-01 06:11:51 [model_runner.py:1127] Graph capturing finished in 5 secs, took 0.13 GiB | |
| (EngineCore pid=235400) INFO 11-01 06:11:51 [gpu_worker.py:971] CUDA graph pool memory: 0.13 GiB (actual), 0.3 GiB (estimated), difference: 0.17 GiB (130.3%). | |
| (EngineCore pid=235400) INFO 11-01 06:11:51 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 5.0 GiB for consumed memory (weights + non-torch), 1.8 GiB for peak activation, and 0.13 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=70691352802` (65.84 GiB) to fit into requested memory, or `--kv-cache-memory=77055522304` (71.76 GiB) to fully utilize gpu memory. Current kv cache memory in use is 66.11 GiB. | |
| (EngineCore pid=235400) INFO 11-01 06:11:51 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=235400) INFO 11-01 06:11:52 [core.py:395] init engine (profile, create kv cache, warmup model) took 51.84 s (compilation: 22.27 s) | |
| (EngineCore pid=235400) INFO 11-01 06:11:52 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=235115) INFO 11-01 06:11:53 [hf.py:648] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=235400) WARNING 11-01 06:11:53 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=235400) WARNING 11-01 06:11:54 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=235400) WARNING 11-01 06:11:55 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=235115) INFO 11-01 06:12:05 [loggers.py:335] Engine 000: Avg prompt throughput: 8740.6 tokens/s, Avg generation throughput: 5753.7 tokens/s, Running: 92 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 11.0% | |
| (MainProcess pid=235115) INFO 11-01 06:12:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2199.6 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 11.0% | |
| (EngineCore pid=235400) WARNING 11-01 06:12:22 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| {"board": "pubmedqa", "closed": 300, "of": 300, "tokens_median": 237.0} | |
| (MainProcess pid=235115) INFO 11-01 06:12:25 [loggers.py:335] Engine 000: Avg prompt throughput: 6629.4 tokens/s, Avg generation throughput: 1453.0 tokens/s, Running: 96 reqs, Waiting: 201 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 57.9% | |
| (MainProcess pid=235115) INFO 11-01 06:12:35 [loggers.py:335] Engine 000: Avg prompt throughput: 12016.3 tokens/s, Avg generation throughput: 5814.3 tokens/s, Running: 96 reqs, Waiting: 6 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 46.7% | |
| {"board": "helpsteer2", "closed": 300, "of": 300, "tokens_median": 263.5} | |
| (MainProcess pid=235115) INFO 11-01 06:12:45 [loggers.py:335] Engine 000: Avg prompt throughput: 5275.5 tokens/s, Avg generation throughput: 3603.7 tokens/s, Running: 96 reqs, Waiting: 197 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 66.7% | |
| (MainProcess pid=235115) INFO 11-01 06:12:55 [loggers.py:335] Engine 000: Avg prompt throughput: 7749.0 tokens/s, Avg generation throughput: 7669.4 tokens/s, Running: 96 reqs, Waiting: 17 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 56.9% | |
| (MainProcess pid=235115) INFO 11-01 06:13:05 [loggers.py:335] Engine 000: Avg prompt throughput: 776.9 tokens/s, Avg generation throughput: 5278.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 55.4% | |
| (MainProcess pid=235115) INFO 11-01 06:13:15 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 722.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 55.4% | |
| {"board": "banking77", "closed": 300, "of": 300, "tokens_median": 366.5} | |
| {"saved": "data/jevals/runs/base_2b_think.json"} | |
| (MainProcess pid=235115) INFO 11-01 06:13:22 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=235400) INFO 11-01 06:13:22 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=235400) INFO 11-01 06:13:22 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=235400) INFO 11-01 06:13:22 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=235400) INFO 11-01 06:13:22 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| FAILED v2_2b_think | |
| (MainProcess pid=246178) INFO 11-01 07:06:05 [api_utils.py:286] non-default args: {'dtype': 'bfloat16', 'seed': 17, 'max_model_len': 32768, 'enable_prefix_caching': True, 'max_num_batched_tokens': 32768, 'max_num_seqs': 256, 'max_logprobs': 26, 'logprobs_mode': 'processed_logprobs', 'model': 'runs/v2/checkpoints/rl40_merged'} | |
| (MainProcess pid=246178) INFO 11-01 07:06:05 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM | |
| (MainProcess pid=246178) INFO 11-01 07:06:05 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=246178) INFO 11-01 07:06:05 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=32768. | |
| (MainProcess pid=246178) INFO 11-01 07:06:05 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled | |
| (MainProcess pid=246178) INFO 11-01 07:06:05 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=246471) INFO 11-01 07:06:13 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='runs/v2/checkpoints/rl40_merged', speculative_config=None, tokenizer='runs/v2/checkpoints/rl40_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=runs/v2/checkpoints/rl40_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=246471) INFO 11-01 07:06:14 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_6a588e5133c241b4bd1825b560cfea6e backend=nccl | |
| (EngineCore pid=246471) INFO 11-01 07:06:15 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=246471) INFO 11-01 07:06:15 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=246471) INFO 11-01 07:06:16 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=246471) INFO 11-01 07:06:16 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=246471) INFO 11-01 07:06:17 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=246471) INFO 11-01 07:06:17 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=246471) INFO 11-01 07:06:18 [base_loader.py:82] Initializing model took 1.390556 seconds | |
| (EngineCore pid=246471) INFO 11-01 07:06:18 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 113.95 GiB. | |
| (EngineCore pid=246471) INFO 11-01 07:06:18 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.11s/it] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.11s/it] | |
| (EngineCore pid=246471) INFO 11-01 07:06:19 [default_loader.py:501] Loading weights took 1.27 seconds | |
| (EngineCore pid=246471) INFO 11-01 07:06:20 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.531713 seconds | |
| (EngineCore pid=246471) INFO 11-01 07:06:20 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=246471) INFO 11-01 07:06:20 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=246471) INFO 11-01 07:06:20 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=246471) INFO 11-01 07:06:20 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=246471) INFO 11-01 07:06:20 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=246471) INFO 11-01 07:06:21 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=246471) INFO 11-01 07:06:27 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/6b7570b37e/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=246471) INFO 11-01 07:06:27 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.35 s | |
| (EngineCore pid=246471) INFO 11-01 07:06:43 [backends.py:398] [backbone] Compiling a graph for compile range (1, 32768) takes 15.94 s | |
| (EngineCore pid=246471) INFO 11-01 07:06:45 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20354119 bytes total | |
| (EngineCore pid=246471) INFO 11-01 07:06:46 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/521d241a2fca701163dd04c49a609eeb1b47ed608dcac4b70c0911a1791c96fa/rank_0_0/model | |
| (EngineCore pid=246471) INFO 11-01 07:06:46 [monitor.py:62] [backbone] torch.compile took 24.26 s in total | |
| (EngineCore pid=246471) INFO 11-01 07:08:13 [monitor.py:91] [backbone] Initial profiling/warmup run took 87.24 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:02<01:50, 2.20s/it] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:02<01:06, 1.36s/it] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:03<00:42, 1.13it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:31, 1.51it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:04<00:27, 1.65it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:27, 1.66it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:05<00:25, 1.71it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:24, 1.79it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:06<00:21, 1.91it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:19, 2.09it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:17, 2.32it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:07<00:15, 2.59it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:13, 2.79it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:12, 3.03it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:10, 3.32it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:08<00:09, 3.68it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:09<00:17, 1.93it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:09<00:14, 2.28it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:09<00:08, 3.86it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:09<00:04, 6.49it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:09<00:02, 9.07it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:10<00:01, 11.45it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:10<00:01, 13.56it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:10<00:01, 14.70it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:10<00:00, 16.19it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:10<00:00, 17.36it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:10<00:00, 18.37it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:10<00:00, 19.16it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:11<00:00, 19.65it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:11<00:00, 4.52it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 26.46it/s] | |
| (EngineCore pid=246471) INFO 11-01 07:08:29 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.40 GiB | |
| (EngineCore pid=246471) INFO 11-01 07:08:30 [gpu_worker.py:759] Available KV cache memory: 58.91 GiB | |
| (EngineCore pid=246471) INFO 11-01 07:08:30 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9140 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9260. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=246471) INFO 11-01 07:08:30 [kv_cache_utils.py:2327] GPU KV cache size: 1,736,229 tokens, Maximum concurrency for 32,768 tokens per request: 52.99x | |
| (EngineCore pid=246471) INFO 11-01 07:08:30 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=246471) INFO 11-01 07:08:30 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=246471) INFO 11-01 07:08:30 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=246471) INFO 11-01 07:08:30 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=246471) INFO 11-01 07:08:30 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=246471) INFO 11-01 07:08:32 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:41, 1.22it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:40, 1.21it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:38, 1.25it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:36, 1.30it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:06<01:12, 1.57s/it] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:06<00:55, 1.24s/it] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:07<00:44, 1.01s/it] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:07<00:36, 1.18it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:08<00:30, 1.39it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:08<00:25, 1.62it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:08<00:21, 1.87it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:09<00:17, 2.18it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:09<00:15, 2.45it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:09<00:13, 2.75it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:09<00:11, 3.14it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:10<00:09, 3.75it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:10<00:04, 7.03it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:10<00:02, 10.12it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:10<00:02, 12.80it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:10<00:01, 15.02it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:10<00:01, 16.79it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:10<00:00, 17.99it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:11<00:00, 19.10it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:11<00:00, 19.89it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:11<00:00, 20.54it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:11<00:00, 21.06it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:11<00:00, 21.50it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:11<00:00, 4.35it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 28.65it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 27.90it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:00, 27.62it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:00, 27.48it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 27.43it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 27.29it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 27.46it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:00<00:00, 27.37it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:00<00:00, 27.72it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 28.05it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 28.20it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 27.85it/s] | |
| (EngineCore pid=246471) INFO 11-01 07:09:05 [model_runner.py:1127] Graph capturing finished in 15 secs, took 0.24 GiB | |
| (EngineCore pid=246471) INFO 11-01 07:09:05 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (99.2%). | |
| (EngineCore pid=246471) INFO 11-01 07:09:05 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 10.52 GiB for consumed memory (weights + non-torch), 3.47 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=62845904098` (58.53 GiB) to fit into requested memory, or `--kv-cache-memory=69210073600` (64.46 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.91 GiB. | |
| (EngineCore pid=246471) INFO 11-01 07:09:06 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=246471) INFO 11-01 07:09:06 [core.py:395] init engine (profile, create kv cache, warmup model) took 166.11 s (compilation: 24.26 s) | |
| (EngineCore pid=246471) INFO 11-01 07:09:06 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=246178) INFO 11-01 07:09:08 [hf.py:648] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. | |
| (EngineCore pid=246471) WARNING 11-01 07:09:08 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (MainProcess pid=246178) INFO 11-01 07:09:20 [loggers.py:335] Engine 000: Avg prompt throughput: 21399.3 tokens/s, Avg generation throughput: 39.2 tokens/s, Running: 34 reqs, Waiting: 46 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 0.2% | |
| (MainProcess pid=246178) INFO 11-01 07:09:30 [loggers.py:335] Engine 000: Avg prompt throughput: 27983.7 tokens/s, Avg generation throughput: 53.4 tokens/s, Running: 68 reqs, Waiting: 349 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.4% | |
| (MainProcess pid=246178) INFO 11-01 07:09:40 [loggers.py:335] Engine 000: Avg prompt throughput: 27105.6 tokens/s, Avg generation throughput: 58.0 tokens/s, Running: 72 reqs, Waiting: 53 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=246178) INFO 11-01 07:09:52 [loggers.py:335] Engine 000: Avg prompt throughput: 26867.0 tokens/s, Avg generation throughput: 57.3 tokens/s, Running: 68 reqs, Waiting: 318 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=246178) INFO 11-01 07:10:03 [loggers.py:335] Engine 000: Avg prompt throughput: 26989.0 tokens/s, Avg generation throughput: 58.1 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=246178) INFO 11-01 07:10:14 [loggers.py:335] Engine 000: Avg prompt throughput: 26711.5 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 70 reqs, Waiting: 247 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=246178) INFO 11-01 07:10:24 [loggers.py:335] Engine 000: Avg prompt throughput: 26949.2 tokens/s, Avg generation throughput: 58.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=246178) INFO 11-01 07:10:35 [loggers.py:335] Engine 000: Avg prompt throughput: 26638.1 tokens/s, Avg generation throughput: 56.3 tokens/s, Running: 70 reqs, Waiting: 246 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0% | |
| (MainProcess pid=246178) INFO 11-01 07:10:45 [loggers.py:335] Engine 000: Avg prompt throughput: 26967.0 tokens/s, Avg generation throughput: 58.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% | |
| {"saved": "data/jevals/runs/rl40.json"} | |
| (MainProcess pid=246178) INFO 11-01 07:10:45 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=246471) INFO 11-01 07:10:45 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=246471) INFO 11-01 07:10:45 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=246471) INFO 11-01 07:10:45 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=246471) INFO 11-01 07:10:45 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
Xet Storage Details
- Size:
- 317 kB
- Xet hash:
- ad2041bdb15f1f22c4614df542a3f15337e55ad9d3b68b25cfe4a74b1aa91fd6
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.