Buckets:
{"time": "2026-10-31T00:49:37Z", "event": "skip", "stage": "prep", "hours_left": 8.0}
{"time": "2026-10-31T00:49:37Z", "event": "start", "stage": "probe", "hours_left": 8.0}
(MainProcess pid=25496) INFO 10-31 00:49:49 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration
(MainProcess pid=25496) INFO 10-31 00:49:49 [model.py:2152] Using max model len 32768
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1166.68it/s]
(MainProcess pid=25496) INFO 10-31 00:49:51 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=25496) INFO 10-31 00:49:51 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.73it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 7.62it/s]
(MainProcess pid=25496) INFO 10-31 00:49:52 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=25496) WARNING 10-31 00:50:03 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
(EngineCore pid=25921) INFO 10-31 00:50:14 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=25921) INFO 10-31 00:50:15 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_1cdc2a918df5485caa3258a5a3b3664c backend=nccl
(EngineCore pid=25921) INFO 10-31 00:50:15 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=25921) INFO 10-31 00:50:15 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=25921) INFO 10-31 00:50:17 [model_runner.py:392] Loading model from scratch...
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1077.81it/s]
(EngineCore pid=25921) INFO 10-31 00:50:18 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=25921) INFO 10-31 00:50:18 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=25921) INFO 10-31 00:50:18 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=25921) INFO 10-31 00:50:19 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=25921) INFO 10-31 00:50:19 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=25921) INFO 10-31 00:50:19 [base_loader.py:82] Initializing model took 2.131808 seconds
(EngineCore pid=25921) INFO 10-31 00:50:19 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.47 GiB.
(EngineCore pid=25921) INFO 10-31 00:50:19 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.67it/s]
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.68it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.68it/s]
(EngineCore pid=25921) INFO 10-31 00:50:20 [default_loader.py:501] Loading weights took 0.75 seconds
(EngineCore pid=25921) INFO 10-31 00:50:21 [model_runner.py:424] Model loading took 8.56 GiB memory and 4.702506 seconds
(EngineCore pid=25921) INFO 10-31 00:50:21 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=25921) INFO 10-31 00:50:21 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=25921) INFO 10-31 00:50:21 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=25921) INFO 10-31 00:50:21 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=25921) INFO 10-31 00:50:21 [utils.py:347] Using LBNHC KV cache layout.
[transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning.
(MainProcess pid=25496) INFO 10-31 00:50:23 [base.py:280] Multi-modal warmup completed in 20.065s
(EngineCore pid=25921) INFO 10-31 00:50:29 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=25921) INFO 10-31 00:50:34 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=25921) INFO 10-31 00:50:39 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/c4af2daa17/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=25921) INFO 10-31 00:50:39 [backends.py:1166] [backbone] Dynamo bytecode transform time: 4.61 s
(EngineCore pid=25921) INFO 10-31 00:50:56 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 17.28 s
(EngineCore pid=25921) INFO 10-31 00:50:59 [backends.py:927] collected artifacts: 33 entries, 13 artifacts, 18863803 bytes total
(EngineCore pid=25921) INFO 10-31 00:50:59 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/cc0f0cae53dc3e1375a6d2ed4c88238808e3367e1d9abaa0419cc3d58ebc1b12/rank_0_0/model
(EngineCore pid=25921) INFO 10-31 00:50:59 [monitor.py:62] [backbone] torch.compile took 24.79 s in total
(EngineCore pid=25921) INFO 10-31 00:52:25 [monitor.py:91] [backbone] Initial profiling/warmup run took 85.73 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:30, 1.12it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:01<00:26, 1.22it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:18, 1.75it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:02<00:15, 2.03it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:02<00:13, 2.24it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:11, 2.63it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:03<00:10, 2.77it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:03<00:08, 3.12it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:08, 3.00it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:04<00:08, 2.95it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:04<00:07, 3.25it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:04<00:07, 3.28it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:06, 3.67it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:07<00:21, 1.02s/it]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:07<00:15, 1.31it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:08<00:11, 1.71it/s]Capturing CUDA graphs (PIECEWISE): 49%|████▊ | 17/35 [00:09<00:12, 1.39it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:09<00:09, 1.76it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:09<00:05, 2.81it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:09<00:03, 4.13it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:09<00:01, 5.73it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:09<00:01, 7.39it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:10<00:00, 8.35it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:10<00:00, 10.09it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:10<00:00, 11.78it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:10<00:00, 13.05it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:10<00:00, 3.31it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 23.26it/s]
(EngineCore pid=25921) INFO 10-31 00:52:39 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.28 GiB
(EngineCore pid=25921) INFO 10-31 00:52:39 [gpu_worker.py:759] Available KV cache memory: 58.36 GiB
(EngineCore pid=25921) INFO 10-31 00:52:39 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=25921) INFO 10-31 00:52:39 [kv_cache_utils.py:2327] GPU KV cache size: 1,719,607 tokens, Maximum concurrency for 32,768 tokens per request: 52.48x
(EngineCore pid=25921) INFO 10-31 00:52:39 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=25921) INFO 10-31 00:52:39 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=25921) INFO 10-31 00:52:39 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=25921) INFO 10-31 00:52:39 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=25921) INFO 10-31 00:52:39 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=25921) INFO 10-31 00:52:41 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)].
(EngineCore pid=25921) INFO 10-31 00:52:41 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:11, 2.99it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.59it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:12, 2.59it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.65it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:10, 2.76it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:10, 2.83it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:09, 3.10it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:08, 3.26it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:08, 2.95it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:07, 3.18it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:07, 3.20it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:06, 3.55it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:06, 3.51it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:05, 3.98it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:05, 4.00it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:04, 4.43it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:02, 6.62it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:05<00:02, 7.38it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:05<00:01, 9.07it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:05<00:01, 10.69it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:05<00:00, 11.95it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:05<00:00, 11.12it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:05<00:00, 12.50it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 13.45it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:06<00:00, 11.06it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:06<00:00, 5.58it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 25.15it/s]Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 25.44it/s]Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 22.67it/s]Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 22.18it/s]Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 23.26it/s]Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 23.42it/s]Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 23.23it/s]
(EngineCore pid=25921) INFO 10-31 00:53:08 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.14 GiB
(EngineCore pid=25921) INFO 10-31 00:53:08 [gpu_worker.py:971] CUDA graph pool memory: 0.14 GiB (actual), 0.32 GiB (estimated), difference: 0.18 GiB (132.9%).
(EngineCore pid=25921) INFO 10-31 00:53:08 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 11.05 GiB for consumed memory (weights + non-torch), 1.92 GiB for peak activation, and 0.14 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=62358280090` (58.08 GiB) to fit into requested memory, or `--kv-cache-memory=70424346112` (65.59 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.36 GiB.
(EngineCore pid=25921) INFO 10-31 00:53:09 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=25921) INFO 10-31 00:53:09 [core.py:395] init engine (profile, create kv cache, warmup model) took 167.63 s (compilation: 24.79 s)
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 3.68it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 7.14it/s]
(EngineCore pid=25921) INFO 10-31 00:53:09 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=25496) WARNING 10-31 00:53:10 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=25921) WARNING 10-31 00:53:10 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=25921) WARNING 10-31 00:53:11 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=25921) WARNING 10-31 00:53:12 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "00:54:09", "elapsed_min": 1.0, "submitted": 128, "finished": 18, "in_flight": 110, "gen_tokens": 57273, "tok_per_s": 955, "chain_tokens_median": 734.0, "chain_tokens_p90": 1303.8, "chain_tokens_max": 2064, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "00:55:09", "elapsed_min": 2.0, "submitted": 128, "finished": 31, "in_flight": 97, "gen_tokens": 160684, "tok_per_s": 1339, "chain_tokens_median": 866.5, "chain_tokens_p90": 2762.9, "chain_tokens_max": 4367, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "00:56:09", "elapsed_min": 3.0, "submitted": 128, "finished": 61, "in_flight": 67, "gen_tokens": 408301, "tok_per_s": 2268, "chain_tokens_median": 969.0, "chain_tokens_p90": 3920.2000000000003, "chain_tokens_max": 6404, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "00:57:09", "elapsed_min": 4.0, "submitted": 128, "finished": 90, "in_flight": 38, "gen_tokens": 744826, "tok_per_s": 3103, "chain_tokens_median": 1202.5, "chain_tokens_p90": 4983.400000000002, "chain_tokens_max": 8258, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "00:58:09", "elapsed_min": 5.0, "submitted": 128, "finished": 111, "in_flight": 17, "gen_tokens": 1030362, "tok_per_s": 3434, "chain_tokens_median": 1366.5, "chain_tokens_p90": 5461.599999999999, "chain_tokens_max": 9949, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "00:59:09", "elapsed_min": 6.0, "submitted": 128, "finished": 124, "in_flight": 4, "gen_tokens": 1338435, "tok_per_s": 3718, "chain_tokens_median": 1763.0, "chain_tokens_p90": 6118.0, "chain_tokens_max": 11487, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:00:09", "elapsed_min": 7.0, "submitted": 128, "finished": 125, "in_flight": 3, "gen_tokens": 1367754, "tok_per_s": 3256, "chain_tokens_median": 1798.5, "chain_tokens_p90": 6196.1, "chain_tokens_max": 11487, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:01:09", "elapsed_min": 8.0, "submitted": 128, "finished": 126, "in_flight": 2, "gen_tokens": 1417464, "tok_per_s": 2953, "chain_tokens_median": 1853.5, "chain_tokens_p90": 6267.4, "chain_tokens_max": 19323, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:02:09", "elapsed_min": 9.0, "submitted": 128, "finished": 126, "in_flight": 2, "gen_tokens": 1417464, "tok_per_s": 2625, "chain_tokens_median": 1853.5, "chain_tokens_p90": 6267.4, "chain_tokens_max": 19323, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=25496) INFO 10-31 01:03:02 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=25496) INFO 10-31 01:03:02 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=25496) INFO 10-31 01:03:02 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=25921) INFO 10-31 01:03:02 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=25921) INFO 10-31 01:03:02 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=25921) INFO 10-31 01:03:02 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=25921) INFO 10-31 01:03:02 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=25496) INFO 10-31 01:03:06 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=25496) INFO 10-31 01:03:06 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=25496) INFO 10-31 01:03:06 [core_client.py:777] [shutdown] MPClient: complete
{
"items": 128,
"samples": 512,
"tokens_total": 1562525,
"tokens_median": 1878.0,
"tokens_p90": 6413.900000000001,
"tokens_max": 32471,
"parse_rate": 0.9921875,
"looped": 4,
"items_with_correct_chain": 111,
"majority_accuracy": 0.8125,
"seconds": 798.2385666370392,
"gen_tok_per_s": 1957.4661828015678,
"engine": {
"submitted": 128,
"finished": 128,
"gen_tokens": 1562525,
"t0": 1793407989.7934816,
"seconds": 592.3575947284698,
"tok_per_s": 2637.806983324396
},
"N_fit": 1587
}
{"time": "2026-10-31T01:03:09Z", "event": "done", "stage": "probe", "hours_left": 7.77}
{"time": "2026-10-31T01:03:26Z", "event": "start", "stage": "baseline", "hours_left": 7.77}
(MainProcess pid=27742) INFO 10-31 01:03:39 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration
(MainProcess pid=27742) INFO 10-31 01:03:39 [model.py:2152] Using max model len 32768
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1315.38it/s]
(MainProcess pid=27742) INFO 10-31 01:03:41 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=27742) INFO 10-31 01:03:41 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.46it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 8.15it/s]
(MainProcess pid=27742) INFO 10-31 01:03:42 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=27742) WARNING 10-31 01:03:53 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
(EngineCore pid=28182) INFO 10-31 01:04:03 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=28182) INFO 10-31 01:04:04 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_633fbdd626484f9c92eeae6e2b5b8bd6 backend=nccl
(EngineCore pid=28182) INFO 10-31 01:04:04 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=28182) INFO 10-31 01:04:04 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=28182) INFO 10-31 01:04:06 [model_runner.py:392] Loading model from scratch...
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1131.02it/s]
(EngineCore pid=28182) INFO 10-31 01:04:07 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=28182) INFO 10-31 01:04:07 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=28182) INFO 10-31 01:04:07 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=28182) INFO 10-31 01:04:08 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=28182) INFO 10-31 01:04:08 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=28182) INFO 10-31 01:04:08 [base_loader.py:82] Initializing model took 2.133625 seconds
(EngineCore pid=28182) INFO 10-31 01:04:08 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.61 GiB.
(EngineCore pid=28182) INFO 10-31 01:04:08 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 1.83it/s]
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=28182) WARNING 10-31 01:04:09 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.94it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.93it/s]
(EngineCore pid=28182) INFO 10-31 01:04:09 [default_loader.py:501] Loading weights took 1.04 seconds
(EngineCore pid=28182) INFO 10-31 01:04:10 [model_runner.py:424] Model loading took 8.56 GiB memory and 5.040389 seconds
(EngineCore pid=28182) INFO 10-31 01:04:11 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=28182) INFO 10-31 01:04:11 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=28182) INFO 10-31 01:04:11 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=28182) INFO 10-31 01:04:11 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=28182) INFO 10-31 01:04:11 [utils.py:347] Using LBNHC KV cache layout.
[transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning.
(MainProcess pid=27742) INFO 10-31 01:04:15 [base.py:280] Multi-modal warmup completed in 21.668s
(EngineCore pid=28182) INFO 10-31 01:04:18 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=28182) INFO 10-31 01:04:23 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=13 num_submods=33
(EngineCore pid=28182) INFO 10-31 01:04:23 [decorators.py:319] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/cc0f0cae53dc3e1375a6d2ed4c88238808e3367e1d9abaa0419cc3d58ebc1b12/rank_0_0/model
(EngineCore pid=28182) INFO 10-31 01:04:23 [monitor.py:62] [backbone] torch.compile took 0.98 s in total
(EngineCore pid=28182) INFO 10-31 01:04:23 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.32 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:16, 2.09it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.64it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:03<00:46, 1.46s/it]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:03<00:31, 1.00s/it]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:04<00:22, 1.35it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:04<00:16, 1.74it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:04<00:13, 2.14it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:04<00:10, 2.52it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:05<00:09, 2.69it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:05<00:08, 2.92it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:05<00:07, 3.09it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:05<00:06, 3.41it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:06<00:05, 3.79it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:06<00:05, 4.17it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:06<00:04, 4.60it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:06<00:03, 5.05it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:06<00:02, 7.27it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:06<00:01, 9.48it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:07<00:01, 11.43it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:07<00:00, 12.58it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:07<00:00, 14.14it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:07<00:00, 15.34it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:07<00:00, 16.39it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:07<00:00, 16.88it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:07<00:00, 17.40it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:07<00:00, 4.52it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 24.25it/s]
(EngineCore pid=28182) INFO 10-31 01:04:33 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.28 GiB
(EngineCore pid=28182) INFO 10-31 01:04:33 [gpu_worker.py:759] Available KV cache memory: 60.55 GiB
(EngineCore pid=28182) INFO 10-31 01:04:33 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=28182) INFO 10-31 01:04:33 [kv_cache_utils.py:2327] GPU KV cache size: 1,784,193 tokens, Maximum concurrency for 32,768 tokens per request: 54.45x
(EngineCore pid=28182) INFO 10-31 01:04:33 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=28182) INFO 10-31 01:04:33 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=28182) INFO 10-31 01:04:33 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=28182) INFO 10-31 01:04:33 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=28182) INFO 10-31 01:04:33 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=28182) INFO 10-31 01:04:33 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)].
(EngineCore pid=28182) INFO 10-31 01:04:33 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:10, 3.35it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.75it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:12, 2.66it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.68it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:10, 2.78it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:09, 2.92it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:08, 3.14it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:08, 3.36it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:02<00:07, 3.28it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:07, 3.13it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:07, 3.32it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:06, 3.62it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:05, 3.81it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:05, 4.16it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:04, 4.55it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:03, 4.97it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:02, 7.09it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:04<00:01, 8.88it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:04<00:01, 10.37it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:05<00:00, 11.63it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:05<00:00, 12.55it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:05<00:00, 13.13it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:05<00:00, 13.68it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 13.90it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:05<00:00, 14.16it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:05<00:00, 6.00it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 20.51it/s]Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 20.71it/s]Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 20.72it/s]Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 20.92it/s]Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 21.39it/s]Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 21.48it/s]Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 21.23it/s]
(EngineCore pid=28182) INFO 10-31 01:04:41 [model_runner.py:1127] Graph capturing finished in 7 secs, took 0.14 GiB
(EngineCore pid=28182) INFO 10-31 01:04:41 [gpu_worker.py:971] CUDA graph pool memory: 0.14 GiB (actual), 0.32 GiB (estimated), difference: 0.18 GiB (132.9%).
(EngineCore pid=28182) INFO 10-31 01:04:41 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 8.86 GiB for consumed memory (weights + non-torch), 1.92 GiB for peak activation, and 0.14 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=64709187482` (60.27 GiB) to fit into requested memory, or `--kv-cache-memory=72775253504` (67.78 GiB) to fully utilize gpu memory. Current kv cache memory in use is 60.55 GiB.
(EngineCore pid=28182) INFO 10-31 01:04:42 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=28182) INFO 10-31 01:04:42 [core.py:395] init engine (profile, create kv cache, warmup model) took 31.63 s (compilation: 0.98 s)
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.44it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 8.25it/s]
(EngineCore pid=28182) INFO 10-31 01:04:43 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=27742) WARNING 10-31 01:04:43 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=28182) WARNING 10-31 01:04:44 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout base: 5957 items in 114s
(EngineCore pid=28182) WARNING 10-31 01:06:37 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=28182) WARNING 10-31 01:06:37 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=28182) WARNING 10-31 01:06:37 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "01:07:37", "elapsed_min": 1.0, "submitted": 329, "finished": 73, "in_flight": 256, "gen_tokens": 84514, "tok_per_s": 1408, "chain_tokens_median": 986.0, "chain_tokens_p90": 2080.8, "chain_tokens_max": 2321, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:08:37", "elapsed_min": 2.0, "submitted": 392, "finished": 136, "in_flight": 256, "gen_tokens": 232486, "tok_per_s": 1937, "chain_tokens_median": 1451.0, "chain_tokens_p90": 3250.5, "chain_tokens_max": 4430, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:09:37", "elapsed_min": 3.0, "submitted": 475, "finished": 219, "in_flight": 256, "gen_tokens": 463478, "tok_per_s": 2575, "chain_tokens_median": 1658.0, "chain_tokens_p90": 4690.8, "chain_tokens_max": 6432, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:10:37", "elapsed_min": 4.0, "submitted": 527, "finished": 271, "in_flight": 256, "gen_tokens": 669475, "tok_per_s": 2789, "chain_tokens_median": 1993.0, "chain_tokens_p90": 5312.0, "chain_tokens_max": 8269, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:11:37", "elapsed_min": 5.0, "submitted": 604, "finished": 348, "in_flight": 256, "gen_tokens": 922303, "tok_per_s": 3074, "chain_tokens_median": 2004.5, "chain_tokens_p90": 5857.300000000001, "chain_tokens_max": 9657, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:12:37", "elapsed_min": 6.0, "submitted": 721, "finished": 465, "in_flight": 256, "gen_tokens": 1182133, "tok_per_s": 3284, "chain_tokens_median": 1734.0, "chain_tokens_p90": 5811.6, "chain_tokens_max": 11244, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:13:37", "elapsed_min": 7.0, "submitted": 805, "finished": 549, "in_flight": 256, "gen_tokens": 1393709, "tok_per_s": 3318, "chain_tokens_median": 1665.0, "chain_tokens_p90": 6047.599999999999, "chain_tokens_max": 13383, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:14:37", "elapsed_min": 8.0, "submitted": 886, "finished": 630, "in_flight": 256, "gen_tokens": 1624499, "tok_per_s": 3384, "chain_tokens_median": 1725.0, "chain_tokens_p90": 6091.800000000003, "chain_tokens_max": 14175, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:15:37", "elapsed_min": 9.0, "submitted": 982, "finished": 726, "in_flight": 256, "gen_tokens": 1874461, "tok_per_s": 3471, "chain_tokens_median": 1661.5, "chain_tokens_p90": 6075.5, "chain_tokens_max": 14175, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:16:37", "elapsed_min": 10.0, "submitted": 1090, "finished": 834, "in_flight": 256, "gen_tokens": 2149312, "tok_per_s": 3582, "chain_tokens_median": 1655.0, "chain_tokens_p90": 5980.500000000002, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:17:37", "elapsed_min": 11.0, "submitted": 1154, "finished": 898, "in_flight": 256, "gen_tokens": 2302889, "tok_per_s": 3489, "chain_tokens_median": 1633.0, "chain_tokens_p90": 5898.100000000002, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:18:37", "elapsed_min": 12.0, "submitted": 1226, "finished": 970, "in_flight": 256, "gen_tokens": 2500454, "tok_per_s": 3473, "chain_tokens_median": 1644.5, "chain_tokens_p90": 5832.5, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:19:37", "elapsed_min": 13.0, "submitted": 1333, "finished": 1077, "in_flight": 256, "gen_tokens": 2750870, "tok_per_s": 3527, "chain_tokens_median": 1616.0, "chain_tokens_p90": 5770.199999999999, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:20:37", "elapsed_min": 14.0, "submitted": 1489, "finished": 1233, "in_flight": 256, "gen_tokens": 2986596, "tok_per_s": 3555, "chain_tokens_median": 1380.0, "chain_tokens_p90": 5705.0, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:21:37", "elapsed_min": 15.0, "submitted": 1593, "finished": 1337, "in_flight": 256, "gen_tokens": 3166310, "tok_per_s": 3518, "chain_tokens_median": 1303.0, "chain_tokens_p90": 5640.800000000001, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:22:37", "elapsed_min": 16.0, "submitted": 1669, "finished": 1413, "in_flight": 256, "gen_tokens": 3331794, "tok_per_s": 3470, "chain_tokens_median": 1323.0, "chain_tokens_p90": 5626.4, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:23:37", "elapsed_min": 17.0, "submitted": 1795, "finished": 1539, "in_flight": 256, "gen_tokens": 3521226, "tok_per_s": 3452, "chain_tokens_median": 1256.0, "chain_tokens_p90": 5478.8, "chain_tokens_max": 14379, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "01:24:37", "elapsed_min": 18.0, "submitted": 1893, "finished": 1637, "in_flight": 256, "gen_tokens": 3733257, "tok_per_s": 3457, "chain_tokens_median": 1230.0, "chain_tokens_p90": 5395.400000000004, "chain_tokens_max": 31330, "parse_rate": 0.9993891264508247, "looped": 1, "minutes_to_deadline": Infinity}
progress {"time": "01:25:37", "elapsed_min": 19.0, "submitted": 1951, "finished": 1695, "in_flight": 256, "gen_tokens": 3964743, "tok_per_s": 3478, "chain_tokens_median": 1257.0, "chain_tokens_p90": 5443.2000000000035, "chain_tokens_max": 31330, "parse_rate": 0.9976401179941002, "looped": 4, "minutes_to_deadline": Infinity}
progress {"time": "01:26:37", "elapsed_min": 20.0, "submitted": 2068, "finished": 1812, "in_flight": 256, "gen_tokens": 4207918, "tok_per_s": 3506, "chain_tokens_median": 1235.5, "chain_tokens_p90": 5349.800000000001, "chain_tokens_max": 31330, "parse_rate": 0.9972406181015453, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:27:37", "elapsed_min": 21.0, "submitted": 2162, "finished": 1906, "in_flight": 256, "gen_tokens": 4368084, "tok_per_s": 3467, "chain_tokens_median": 1204.0, "chain_tokens_p90": 5249.5, "chain_tokens_max": 31330, "parse_rate": 0.9973767051416579, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:28:37", "elapsed_min": 22.0, "submitted": 2202, "finished": 1946, "in_flight": 256, "gen_tokens": 4501259, "tok_per_s": 3410, "chain_tokens_median": 1231.5, "chain_tokens_p90": 5275.0, "chain_tokens_max": 31330, "parse_rate": 0.9974306269270298, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:29:37", "elapsed_min": 23.0, "submitted": 2277, "finished": 2021, "in_flight": 256, "gen_tokens": 4648414, "tok_per_s": 3368, "chain_tokens_median": 1222.0, "chain_tokens_p90": 5258.0, "chain_tokens_max": 31330, "parse_rate": 0.9975259772389906, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:30:37", "elapsed_min": 24.0, "submitted": 2354, "finished": 2098, "in_flight": 256, "gen_tokens": 4840450, "tok_per_s": 3361, "chain_tokens_median": 1213.0, "chain_tokens_p90": 5244.6, "chain_tokens_max": 32107, "parse_rate": 0.9976167778836987, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:31:37", "elapsed_min": 25.0, "submitted": 2429, "finished": 2173, "in_flight": 256, "gen_tokens": 5030339, "tok_per_s": 3353, "chain_tokens_median": 1218.0, "chain_tokens_p90": 5329.0, "chain_tokens_max": 32107, "parse_rate": 0.9976990335941095, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:32:37", "elapsed_min": 26.0, "submitted": 2506, "finished": 2250, "in_flight": 256, "gen_tokens": 5174005, "tok_per_s": 3317, "chain_tokens_median": 1192.0, "chain_tokens_p90": 5330.9000000000015, "chain_tokens_max": 32107, "parse_rate": 0.9977777777777778, "looped": 5, "minutes_to_deadline": Infinity}
progress {"time": "01:33:37", "elapsed_min": 27.0, "submitted": 2577, "finished": 2321, "in_flight": 256, "gen_tokens": 5404466, "tok_per_s": 3336, "chain_tokens_median": 1193.0, "chain_tokens_p90": 5355.0, "chain_tokens_max": 32654, "parse_rate": 0.996984058595433, "looped": 7, "minutes_to_deadline": Infinity}
progress {"time": "01:34:37", "elapsed_min": 28.0, "submitted": 2680, "finished": 2424, "in_flight": 256, "gen_tokens": 5641858, "tok_per_s": 3358, "chain_tokens_median": 1182.5, "chain_tokens_p90": 5377.700000000001, "chain_tokens_max": 32654, "parse_rate": 0.9966996699669967, "looped": 8, "minutes_to_deadline": Infinity}
progress {"time": "01:35:37", "elapsed_min": 29.0, "submitted": 2736, "finished": 2480, "in_flight": 256, "gen_tokens": 5821732, "tok_per_s": 3346, "chain_tokens_median": 1193.0, "chain_tokens_p90": 5557.1, "chain_tokens_max": 32654, "parse_rate": 0.9967741935483871, "looped": 8, "minutes_to_deadline": Infinity}
progress {"time": "01:36:37", "elapsed_min": 30.0, "submitted": 2801, "finished": 2545, "in_flight": 256, "gen_tokens": 6015310, "tok_per_s": 3342, "chain_tokens_median": 1205.0, "chain_tokens_p90": 5561.599999999999, "chain_tokens_max": 32654, "parse_rate": 0.9964636542239685, "looped": 9, "minutes_to_deadline": Infinity}
progress {"time": "01:37:37", "elapsed_min": 31.0, "submitted": 2845, "finished": 2589, "in_flight": 256, "gen_tokens": 6159155, "tok_per_s": 3311, "chain_tokens_median": 1218.0, "chain_tokens_p90": 5564.400000000001, "chain_tokens_max": 32654, "parse_rate": 0.9961375048281189, "looped": 10, "minutes_to_deadline": Infinity}
progress {"time": "01:38:37", "elapsed_min": 32.0, "submitted": 2916, "finished": 2660, "in_flight": 256, "gen_tokens": 6339467, "tok_per_s": 3302, "chain_tokens_median": 1205.5, "chain_tokens_p90": 5579.0, "chain_tokens_max": 32654, "parse_rate": 0.9958646616541353, "looped": 11, "minutes_to_deadline": Infinity}
progress {"time": "01:39:37", "elapsed_min": 33.0, "submitted": 3019, "finished": 2763, "in_flight": 256, "gen_tokens": 6580922, "tok_per_s": 3324, "chain_tokens_median": 1193.0, "chain_tokens_p90": 5583.000000000001, "chain_tokens_max": 32654, "parse_rate": 0.995656894679696, "looped": 12, "minutes_to_deadline": Infinity}
progress {"time": "01:40:37", "elapsed_min": 34.0, "submitted": 3205, "finished": 2949, "in_flight": 256, "gen_tokens": 6837377, "tok_per_s": 3352, "chain_tokens_median": 1131.0, "chain_tokens_p90": 5557.200000000001, "chain_tokens_max": 32654, "parse_rate": 0.9959308240081384, "looped": 12, "minutes_to_deadline": Infinity}
progress {"time": "01:41:37", "elapsed_min": 35.0, "submitted": 3340, "finished": 3084, "in_flight": 256, "gen_tokens": 7119072, "tok_per_s": 3390, "chain_tokens_median": 1097.5, "chain_tokens_p90": 5564.700000000001, "chain_tokens_max": 32654, "parse_rate": 0.9954604409857328, "looped": 14, "minutes_to_deadline": Infinity}
progress {"time": "01:42:37", "elapsed_min": 36.0, "submitted": 3431, "finished": 3175, "in_flight": 256, "gen_tokens": 7307967, "tok_per_s": 3383, "chain_tokens_median": 1098.0, "chain_tokens_p90": 5533.6, "chain_tokens_max": 32654, "parse_rate": 0.9952755905511811, "looped": 15, "minutes_to_deadline": Infinity}
progress {"time": "01:43:37", "elapsed_min": 37.0, "submitted": 3506, "finished": 3250, "in_flight": 256, "gen_tokens": 7493006, "tok_per_s": 3375, "chain_tokens_median": 1120.5, "chain_tokens_p90": 5522.299999999999, "chain_tokens_max": 32654, "parse_rate": 0.9953846153846154, "looped": 15, "minutes_to_deadline": Infinity}
progress {"time": "01:44:37", "elapsed_min": 38.0, "submitted": 3552, "finished": 3296, "in_flight": 256, "gen_tokens": 7660652, "tok_per_s": 3360, "chain_tokens_median": 1134.5, "chain_tokens_p90": 5499.5, "chain_tokens_max": 32654, "parse_rate": 0.9951456310679612, "looped": 16, "minutes_to_deadline": Infinity}
progress {"time": "01:45:37", "elapsed_min": 39.0, "submitted": 3603, "finished": 3347, "in_flight": 256, "gen_tokens": 7912924, "tok_per_s": 3381, "chain_tokens_median": 1164.0, "chain_tokens_p90": 5543.200000000003, "chain_tokens_max": 32654, "parse_rate": 0.9946220495966537, "looped": 18, "minutes_to_deadline": Infinity}
progress {"time": "01:46:37", "elapsed_min": 40.0, "submitted": 3654, "finished": 3398, "in_flight": 256, "gen_tokens": 8138857, "tok_per_s": 3391, "chain_tokens_median": 1185.0, "chain_tokens_p90": 5565.3, "chain_tokens_max": 32654, "parse_rate": 0.994408475573867, "looped": 19, "minutes_to_deadline": Infinity}
progress {"time": "01:47:37", "elapsed_min": 41.0, "submitted": 3712, "finished": 3456, "in_flight": 256, "gen_tokens": 8343448, "tok_per_s": 3391, "chain_tokens_median": 1203.0, "chain_tokens_p90": 5565.5, "chain_tokens_max": 32654, "parse_rate": 0.9945023148148148, "looped": 19, "minutes_to_deadline": Infinity}
progress {"time": "01:48:37", "elapsed_min": 42.0, "submitted": 3762, "finished": 3506, "in_flight": 256, "gen_tokens": 8531646, "tok_per_s": 3385, "chain_tokens_median": 1218.5, "chain_tokens_p90": 5588.0, "chain_tokens_max": 32654, "parse_rate": 0.9945807187678266, "looped": 19, "minutes_to_deadline": Infinity}
progress {"time": "01:49:37", "elapsed_min": 43.0, "submitted": 3815, "finished": 3559, "in_flight": 256, "gen_tokens": 8708509, "tok_per_s": 3375, "chain_tokens_median": 1233.0, "chain_tokens_p90": 5606.400000000001, "chain_tokens_max": 32654, "parse_rate": 0.9946614217476819, "looped": 19, "minutes_to_deadline": Infinity}
progress {"time": "01:50:37", "elapsed_min": 44.0, "submitted": 3867, "finished": 3611, "in_flight": 256, "gen_tokens": 9012981, "tok_per_s": 3414, "chain_tokens_median": 1255.0, "chain_tokens_p90": 5655.0, "chain_tokens_max": 32654, "parse_rate": 0.993907504846303, "looped": 22, "minutes_to_deadline": Infinity}
progress {"time": "01:51:37", "elapsed_min": 45.0, "submitted": 3908, "finished": 3652, "in_flight": 256, "gen_tokens": 9160768, "tok_per_s": 3393, "chain_tokens_median": 1267.5, "chain_tokens_p90": 5686.700000000001, "chain_tokens_max": 32654, "parse_rate": 0.9939759036144579, "looped": 22, "minutes_to_deadline": Infinity}
progress {"time": "01:52:37", "elapsed_min": 46.0, "submitted": 3971, "finished": 3715, "in_flight": 256, "gen_tokens": 9411451, "tok_per_s": 3410, "chain_tokens_median": 1299.0, "chain_tokens_p90": 5724.4, "chain_tokens_max": 32654, "parse_rate": 0.9940780619111709, "looped": 22, "minutes_to_deadline": Infinity}
progress {"time": "01:53:37", "elapsed_min": 47.0, "submitted": 4028, "finished": 3772, "in_flight": 256, "gen_tokens": 9620049, "tok_per_s": 3411, "chain_tokens_median": 1324.5, "chain_tokens_p90": 5735.5, "chain_tokens_max": 32654, "parse_rate": 0.9941675503711559, "looped": 22, "minutes_to_deadline": Infinity}
progress {"time": "01:54:37", "elapsed_min": 48.0, "submitted": 4089, "finished": 3833, "in_flight": 256, "gen_tokens": 9871397, "tok_per_s": 3427, "chain_tokens_median": 1347.0, "chain_tokens_p90": 5780.800000000002, "chain_tokens_max": 32654, "parse_rate": 0.993999478215497, "looped": 23, "minutes_to_deadline": Infinity}
progress {"time": "01:55:37", "elapsed_min": 49.0, "submitted": 4148, "finished": 3892, "in_flight": 256, "gen_tokens": 10061683, "tok_per_s": 3422, "chain_tokens_median": 1359.5, "chain_tokens_p90": 5790.8, "chain_tokens_max": 32654, "parse_rate": 0.9940904419321686, "looped": 23, "minutes_to_deadline": Infinity}
progress {"time": "01:56:37", "elapsed_min": 50.0, "submitted": 4224, "finished": 3968, "in_flight": 256, "gen_tokens": 10315378, "tok_per_s": 3438, "chain_tokens_median": 1373.0, "chain_tokens_p90": 5775.300000000002, "chain_tokens_max": 32654, "parse_rate": 0.9939516129032258, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "01:57:37", "elapsed_min": 51.0, "submitted": 4298, "finished": 4042, "in_flight": 256, "gen_tokens": 10507533, "tok_per_s": 3434, "chain_tokens_median": 1381.0, "chain_tokens_p90": 5758.000000000001, "chain_tokens_max": 32654, "parse_rate": 0.9940623453735774, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "01:58:37", "elapsed_min": 52.0, "submitted": 4367, "finished": 4111, "in_flight": 256, "gen_tokens": 10696619, "tok_per_s": 3428, "chain_tokens_median": 1395.0, "chain_tokens_p90": 5749.0, "chain_tokens_max": 32654, "parse_rate": 0.9941620043784967, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "01:59:37", "elapsed_min": 53.0, "submitted": 4452, "finished": 4196, "in_flight": 256, "gen_tokens": 10923622, "tok_per_s": 3435, "chain_tokens_median": 1401.0, "chain_tokens_p90": 5736.5, "chain_tokens_max": 32654, "parse_rate": 0.994280266920877, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "02:00:37", "elapsed_min": 54.0, "submitted": 4525, "finished": 4269, "in_flight": 256, "gen_tokens": 11115102, "tok_per_s": 3430, "chain_tokens_median": 1404.0, "chain_tokens_p90": 5730.200000000001, "chain_tokens_max": 32654, "parse_rate": 0.994378074490513, "looped": 24, "minutes_to_deadline": Infinity}
progress {"time": "02:01:37", "elapsed_min": 55.0, "submitted": 4588, "finished": 4332, "in_flight": 256, "gen_tokens": 11295827, "tok_per_s": 3423, "chain_tokens_median": 1410.5, "chain_tokens_p90": 5721.9, "chain_tokens_max": 32654, "parse_rate": 0.9942289935364728, "looped": 25, "minutes_to_deadline": Infinity}
progress {"time": "02:02:37", "elapsed_min": 56.0, "submitted": 4666, "finished": 4410, "in_flight": 256, "gen_tokens": 11529493, "tok_per_s": 3431, "chain_tokens_median": 1421.5, "chain_tokens_p90": 5706.799999999999, "chain_tokens_max": 32654, "parse_rate": 0.9941043083900227, "looped": 26, "minutes_to_deadline": Infinity}
progress {"time": "02:03:37", "elapsed_min": 57.0, "submitted": 4731, "finished": 4475, "in_flight": 256, "gen_tokens": 11730013, "tok_per_s": 3430, "chain_tokens_median": 1441.0, "chain_tokens_p90": 5704.0, "chain_tokens_max": 32654, "parse_rate": 0.9939664804469274, "looped": 27, "minutes_to_deadline": Infinity}
progress {"time": "02:04:37", "elapsed_min": 58.0, "submitted": 4801, "finished": 4545, "in_flight": 256, "gen_tokens": 11922399, "tok_per_s": 3426, "chain_tokens_median": 1448.0, "chain_tokens_p90": 5700.6, "chain_tokens_max": 32654, "parse_rate": 0.994059405940594, "looped": 27, "minutes_to_deadline": Infinity}
progress {"time": "02:05:37", "elapsed_min": 59.0, "submitted": 4877, "finished": 4621, "in_flight": 256, "gen_tokens": 12171920, "tok_per_s": 3438, "chain_tokens_median": 1455.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9937243020991128, "looped": 29, "minutes_to_deadline": Infinity}
progress {"time": "02:06:37", "elapsed_min": 60.0, "submitted": 4942, "finished": 4686, "in_flight": 256, "gen_tokens": 12374714, "tok_per_s": 3437, "chain_tokens_median": 1468.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9935979513444302, "looped": 30, "minutes_to_deadline": Infinity}
progress {"time": "02:07:37", "elapsed_min": 61.0, "submitted": 5007, "finished": 4751, "in_flight": 256, "gen_tokens": 12604284, "tok_per_s": 3444, "chain_tokens_median": 1483.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9932645758787624, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:08:37", "elapsed_min": 62.0, "submitted": 5081, "finished": 4825, "in_flight": 256, "gen_tokens": 12808839, "tok_per_s": 3443, "chain_tokens_median": 1499.0, "chain_tokens_p90": 5682.800000000005, "chain_tokens_max": 32654, "parse_rate": 0.9933678756476684, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:09:37", "elapsed_min": 63.0, "submitted": 5153, "finished": 4897, "in_flight": 256, "gen_tokens": 13030924, "tok_per_s": 3447, "chain_tokens_median": 1514.0, "chain_tokens_p90": 5690.000000000003, "chain_tokens_max": 32654, "parse_rate": 0.9934653869716152, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:10:37", "elapsed_min": 64.0, "submitted": 5209, "finished": 4953, "in_flight": 256, "gen_tokens": 13176545, "tok_per_s": 3431, "chain_tokens_median": 1515.0, "chain_tokens_p90": 5685.400000000002, "chain_tokens_max": 32654, "parse_rate": 0.9935392691298203, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:11:37", "elapsed_min": 65.0, "submitted": 5276, "finished": 5020, "in_flight": 256, "gen_tokens": 13376782, "tok_per_s": 3430, "chain_tokens_median": 1530.5, "chain_tokens_p90": 5676.300000000005, "chain_tokens_max": 32654, "parse_rate": 0.9936254980079682, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:12:37", "elapsed_min": 66.0, "submitted": 5338, "finished": 5082, "in_flight": 256, "gen_tokens": 13567484, "tok_per_s": 3426, "chain_tokens_median": 1538.0, "chain_tokens_p90": 5674.800000000001, "chain_tokens_max": 32654, "parse_rate": 0.9937032664305392, "looped": 32, "minutes_to_deadline": Infinity}
progress {"time": "02:13:37", "elapsed_min": 67.0, "submitted": 5391, "finished": 5135, "in_flight": 256, "gen_tokens": 13739879, "tok_per_s": 3418, "chain_tokens_median": 1546.0, "chain_tokens_p90": 5674.200000000001, "chain_tokens_max": 32654, "parse_rate": 0.9935735150925025, "looped": 33, "minutes_to_deadline": Infinity}
progress {"time": "02:14:37", "elapsed_min": 68.0, "submitted": 5451, "finished": 5195, "in_flight": 256, "gen_tokens": 13950313, "tok_per_s": 3419, "chain_tokens_median": 1560.0, "chain_tokens_p90": 5672.6, "chain_tokens_max": 32654, "parse_rate": 0.9934552454282964, "looped": 34, "minutes_to_deadline": Infinity}
progress {"time": "02:15:37", "elapsed_min": 69.0, "submitted": 5513, "finished": 5257, "in_flight": 256, "gen_tokens": 14156336, "tok_per_s": 3419, "chain_tokens_median": 1575.0, "chain_tokens_p90": 5680.200000000007, "chain_tokens_max": 32654, "parse_rate": 0.9935324329465475, "looped": 34, "minutes_to_deadline": Infinity}
progress {"time": "02:16:37", "elapsed_min": 70.0, "submitted": 5570, "finished": 5314, "in_flight": 256, "gen_tokens": 14364565, "tok_per_s": 3420, "chain_tokens_median": 1581.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9932254422280767, "looped": 36, "minutes_to_deadline": Infinity}
progress {"time": "02:17:37", "elapsed_min": 71.0, "submitted": 5636, "finished": 5380, "in_flight": 256, "gen_tokens": 14593072, "tok_per_s": 3425, "chain_tokens_median": 1586.0, "chain_tokens_p90": 5676.300000000005, "chain_tokens_max": 32654, "parse_rate": 0.9931226765799257, "looped": 37, "minutes_to_deadline": Infinity}
progress {"time": "02:18:37", "elapsed_min": 72.0, "submitted": 5701, "finished": 5445, "in_flight": 256, "gen_tokens": 14830622, "tok_per_s": 3433, "chain_tokens_median": 1589.0, "chain_tokens_p90": 5680.400000000003, "chain_tokens_max": 32654, "parse_rate": 0.9928374655647383, "looped": 39, "minutes_to_deadline": Infinity}
progress {"time": "02:19:37", "elapsed_min": 73.0, "submitted": 5759, "finished": 5503, "in_flight": 256, "gen_tokens": 15008970, "tok_per_s": 3427, "chain_tokens_median": 1591.0, "chain_tokens_p90": 5672.8, "chain_tokens_max": 32654, "parse_rate": 0.9927312375068145, "looped": 40, "minutes_to_deadline": Infinity}
progress {"time": "02:20:37", "elapsed_min": 74.0, "submitted": 5832, "finished": 5576, "in_flight": 256, "gen_tokens": 15200373, "tok_per_s": 3423, "chain_tokens_median": 1590.5, "chain_tokens_p90": 5668.0, "chain_tokens_max": 32654, "parse_rate": 0.9926470588235294, "looped": 41, "minutes_to_deadline": Infinity}
progress {"time": "02:21:37", "elapsed_min": 75.0, "submitted": 5904, "finished": 5648, "in_flight": 256, "gen_tokens": 15428462, "tok_per_s": 3428, "chain_tokens_median": 1592.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9925637393767706, "looped": 42, "minutes_to_deadline": Infinity}
progress {"time": "02:22:37", "elapsed_min": 76.0, "submitted": 5957, "finished": 5712, "in_flight": 245, "gen_tokens": 15584320, "tok_per_s": 3417, "chain_tokens_median": 1591.5, "chain_tokens_p90": 5687.600000000002, "chain_tokens_max": 32654, "parse_rate": 0.9926470588235294, "looped": 42, "minutes_to_deadline": Infinity}
progress {"time": "02:23:37", "elapsed_min": 77.0, "submitted": 5957, "finished": 5783, "in_flight": 174, "gen_tokens": 15739461, "tok_per_s": 3407, "chain_tokens_median": 1587.0, "chain_tokens_p90": 5674.6, "chain_tokens_max": 32654, "parse_rate": 0.9927373335638942, "looped": 42, "minutes_to_deadline": Infinity}
progress {"time": "02:24:37", "elapsed_min": 78.0, "submitted": 5957, "finished": 5841, "in_flight": 116, "gen_tokens": 15895616, "tok_per_s": 3396, "chain_tokens_median": 1586.0, "chain_tokens_p90": 5688.0, "chain_tokens_max": 32654, "parse_rate": 0.9928094504365691, "looped": 42, "minutes_to_deadline": Infinity}
progress {"time": "02:25:37", "elapsed_min": 79.0, "submitted": 5957, "finished": 5903, "in_flight": 54, "gen_tokens": 16243669, "tok_per_s": 3427, "chain_tokens_median": 1601.0, "chain_tokens_p90": 5721.8, "chain_tokens_max": 32654, "parse_rate": 0.9923767575808911, "looped": 45, "minutes_to_deadline": Infinity}
progress {"time": "02:26:37", "elapsed_min": 80.0, "submitted": 5957, "finished": 5938, "in_flight": 19, "gen_tokens": 16513426, "tok_per_s": 3440, "chain_tokens_median": 1615.0, "chain_tokens_p90": 5749.6, "chain_tokens_max": 32654, "parse_rate": 0.9920848770629842, "looped": 47, "minutes_to_deadline": Infinity}
progress {"time": "02:27:37", "elapsed_min": 81.0, "submitted": 5957, "finished": 5948, "in_flight": 9, "gen_tokens": 16651513, "tok_per_s": 3426, "chain_tokens_median": 1619.0, "chain_tokens_p90": 5769.900000000001, "chain_tokens_max": 32654, "parse_rate": 0.9917619367854741, "looped": 49, "minutes_to_deadline": Infinity}
progress {"time": "02:28:37", "elapsed_min": 82.0, "submitted": 5957, "finished": 5950, "in_flight": 7, "gen_tokens": 16716368, "tok_per_s": 3397, "chain_tokens_median": 1621.5, "chain_tokens_p90": 5777.500000000002, "chain_tokens_max": 32654, "parse_rate": 0.9914285714285714, "looped": 51, "minutes_to_deadline": Infinity}
progress {"time": "02:29:37", "elapsed_min": 83.0, "submitted": 5957, "finished": 5952, "in_flight": 5, "gen_tokens": 16780064, "tok_per_s": 3369, "chain_tokens_median": 1622.5, "chain_tokens_p90": 5782.900000000001, "chain_tokens_max": 32654, "parse_rate": 0.9910954301075269, "looped": 53, "minutes_to_deadline": Infinity}
progress {"time": "02:30:37", "elapsed_min": 84.0, "submitted": 5957, "finished": 5953, "in_flight": 4, "gen_tokens": 16812277, "tok_per_s": 3336, "chain_tokens_median": 1623.0, "chain_tokens_p90": 5784.6, "chain_tokens_max": 32654, "parse_rate": 0.9909289433898875, "looped": 54, "minutes_to_deadline": Infinity}
reason base: {"submitted": 5957, "finished": 5957, "gen_tokens": 16942043, "t0": 1793408797.1839204, "seconds": 5098.851123094559, "tok_per_s": 3322.717724246409}
reason base: 5957 items, post-readout in 5890s total
latency base: {"readout_ms": {"median": 42.97343268990517, "p90": 76.37803927063942, "n": 100}, "reasoning_ms": {"median": 10051.338203251362, "p90": 41781.982911750674, "n": 20}}
(MainProcess pid=27742) INFO 10-31 02:50:45 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=27742) INFO 10-31 02:50:45 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=27742) INFO 10-31 02:50:45 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=28182) INFO 10-31 02:50:45 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=28182) INFO 10-31 02:50:45 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=28182) INFO 10-31 02:50:45 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=28182) INFO 10-31 02:50:45 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=27742) INFO 10-31 02:50:49 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=27742) INFO 10-31 02:50:49 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=27742) INFO 10-31 02:50:49 [core_client.py:777] [shutdown] MPClient: complete
{"time": "2026-10-31T02:50:53Z", "event": "done", "stage": "baseline", "hours_left": 5.98}
{"time": "2026-10-31T02:51:11Z", "event": "start", "stage": "generate", "hours_left": 5.97}
{"to_generate": 17823, "already_done": 128, "minutes_left": 130.20927441517512}
(MainProcess pid=32715) INFO 10-31 02:51:24 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration
(MainProcess pid=32715) INFO 10-31 02:51:24 [model.py:2152] Using max model len 32768
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1209.26it/s]
(MainProcess pid=32715) INFO 10-31 02:51:27 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=4096.
(MainProcess pid=32715) INFO 10-31 02:51:27 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.51it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 6.77it/s]
(MainProcess pid=32715) INFO 10-31 02:51:27 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=32715) WARNING 10-31 02:51:39 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
(EngineCore pid=33144) INFO 10-31 02:51:49 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=33144) INFO 10-31 02:51:51 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_4dc4c53d0d1649a08d76719d18d1fba3 backend=nccl
(EngineCore pid=33144) INFO 10-31 02:51:51 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=33144) INFO 10-31 02:51:51 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=33144) INFO 10-31 02:51:52 [model_runner.py:392] Loading model from scratch...
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1022.63it/s]
(EngineCore pid=33144) INFO 10-31 02:51:53 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=33144) INFO 10-31 02:51:53 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=33144) INFO 10-31 02:51:53 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=33144) INFO 10-31 02:51:54 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=33144) INFO 10-31 02:51:54 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=33144) INFO 10-31 02:51:57 [base_loader.py:82] Initializing model took 5.253679 seconds
(EngineCore pid=33144) INFO 10-31 02:51:58 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.62 GiB.
(EngineCore pid=33144) INFO 10-31 02:51:58 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.53it/s]
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=33144) WARNING 10-31 02:51:58 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.59it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.58it/s]
(EngineCore pid=33144) INFO 10-31 02:51:58 [default_loader.py:501] Loading weights took 0.78 seconds
[transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning.
(MainProcess pid=32715) INFO 10-31 02:51:59 [base.py:280] Multi-modal warmup completed in 20.728s
(EngineCore pid=33144) INFO 10-31 02:52:00 [model_runner.py:424] Model loading took 8.56 GiB memory and 8.143870 seconds
(EngineCore pid=33144) INFO 10-31 02:52:00 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=33144) INFO 10-31 02:52:00 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=33144) INFO 10-31 02:52:00 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=33144) INFO 10-31 02:52:00 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=33144) INFO 10-31 02:52:00 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=33144) INFO 10-31 02:52:08 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=33144) INFO 10-31 02:52:11 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=33144) INFO 10-31 02:52:16 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/cb9a0bc415/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=33144) INFO 10-31 02:52:16 [backends.py:1166] [backbone] Dynamo bytecode transform time: 4.62 s
(EngineCore pid=33144) INFO 10-31 02:52:32 [backends.py:398] [backbone] Compiling a graph for compile range (1, 4096) takes 16.41 s
(EngineCore pid=33144) INFO 10-31 02:52:35 [backends.py:927] collected artifacts: 33 entries, 13 artifacts, 18908777 bytes total
(EngineCore pid=33144) INFO 10-31 02:52:35 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/0ec029bac608df5cf30f9ac991aa17123c89f1db43b38d77e398d90dc2716034/rank_0_0/model
(EngineCore pid=33144) INFO 10-31 02:52:35 [monitor.py:62] [backbone] torch.compile took 23.99 s in total
(EngineCore pid=33144) INFO 10-31 02:54:01 [monitor.py:91] [backbone] Initial profiling/warmup run took 85.50 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:01<01:02, 1.25s/it]Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:31, 1.56it/s]Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:01<00:13, 3.54it/s]Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:01<00:08, 5.58it/s]Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:02<00:06, 6.15it/s]Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:02<00:05, 7.98it/s]Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:02<00:03, 9.83it/s]Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:02<00:03, 11.41it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:02<00:03, 9.91it/s]Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:03<00:04, 7.43it/s]Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:03<00:03, 8.44it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:03<00:02, 10.09it/s]Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:03<00:02, 11.53it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:03<00:02, 9.52it/s]Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:03<00:02, 10.24it/s]Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:03<00:01, 11.65it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:04<00:01, 10.82it/s]Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:04<00:01, 12.19it/s]Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:04<00:01, 12.71it/s]Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:07<00:05, 2.23it/s]Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:07<00:03, 3.00it/s]Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:07<00:02, 4.00it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:07<00:01, 5.20it/s]Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:07<00:00, 6.02it/s]Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:07<00:00, 7.50it/s]Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:07<00:00, 9.04it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:08<00:00, 6.26it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 18.47it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 18.43it/s]
(EngineCore pid=33144) INFO 10-31 02:54:12 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.40 GiB
(EngineCore pid=33144) INFO 10-31 02:54:13 [gpu_worker.py:759] Available KV cache memory: 57.76 GiB
(EngineCore pid=33144) INFO 10-31 02:54:13 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8934 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9066. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=33144) INFO 10-31 02:54:13 [kv_cache_utils.py:2327] GPU KV cache size: 1,702,036 tokens, Maximum concurrency for 32,768 tokens per request: 51.94x
(EngineCore pid=33144) INFO 10-31 02:54:13 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=33144) INFO 10-31 02:54:13 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=33144) INFO 10-31 02:54:13 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=33144) INFO 10-31 02:54:13 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=33144) INFO 10-31 02:54:13 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=33144) INFO 10-31 02:54:16 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)].
(EngineCore pid=33144) INFO 10-31 02:54:16 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:05, 9.17it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:00<00:03, 12.30it/s]Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:00<00:03, 13.68it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:00<00:03, 14.25it/s]Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:00<00:02, 14.70it/s]Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:00<00:02, 14.87it/s]Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:00<00:02, 14.23it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:01<00:02, 14.82it/s]Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:01<00:02, 15.18it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:01<00:02, 15.50it/s]Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:01<00:01, 15.66it/s]Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:01<00:01, 15.89it/s]Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:01<00:01, 15.78it/s]Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:01<00:01, 15.97it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:01<00:01, 13.97it/s]Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:02<00:01, 14.58it/s]Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:02<00:01, 14.93it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:02<00:01, 15.33it/s]Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:02<00:00, 15.60it/s]Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:02<00:00, 15.72it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:02<00:00, 15.82it/s]Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:02<00:00, 15.98it/s]Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:02<00:00, 16.12it/s]Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:03<00:00, 16.22it/s]Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:03<00:00, 12.86it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:05<00:00, 2.26it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:05<00:00, 8.64it/s]
Capturing CUDA graphs (FULL): 0%| | 0/51 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 4%|▍ | 2/51 [00:01<00:47, 1.04it/s]Capturing CUDA graphs (FULL): 6%|▌ | 3/51 [00:02<00:40, 1.18it/s]Capturing CUDA graphs (FULL): 10%|▉ | 5/51 [00:02<00:19, 2.40it/s]Capturing CUDA graphs (FULL): 16%|█▌ | 8/51 [00:02<00:09, 4.62it/s]Capturing CUDA graphs (FULL): 22%|██▏ | 11/51 [00:02<00:05, 7.08it/s]Capturing CUDA graphs (FULL): 27%|██▋ | 14/51 [00:03<00:03, 9.53it/s]Capturing CUDA graphs (FULL): 33%|███▎ | 17/51 [00:03<00:02, 12.10it/s]Capturing CUDA graphs (FULL): 39%|███▉ | 20/51 [00:03<00:02, 14.25it/s]Capturing CUDA graphs (FULL): 45%|████▌ | 23/51 [00:03<00:01, 16.34it/s]Capturing CUDA graphs (FULL): 51%|█████ | 26/51 [00:03<00:01, 18.17it/s]Capturing CUDA graphs (FULL): 57%|█████▋ | 29/51 [00:03<00:01, 19.33it/s]Capturing CUDA graphs (FULL): 63%|██████▎ | 32/51 [00:03<00:00, 20.72it/s]Capturing CUDA graphs (FULL): 69%|██████▊ | 35/51 [00:04<00:00, 21.60it/s]Capturing CUDA graphs (FULL): 75%|███████▍ | 38/51 [00:04<00:00, 22.42it/s]Capturing CUDA graphs (FULL): 80%|████████ | 41/51 [00:04<00:00, 22.71it/s]Capturing CUDA graphs (FULL): 86%|████████▋ | 44/51 [00:04<00:00, 23.02it/s]Capturing CUDA graphs (FULL): 92%|█████████▏| 47/51 [00:04<00:00, 23.75it/s]Capturing CUDA graphs (FULL): 98%|█████████▊| 50/51 [00:04<00:00, 24.26it/s]Capturing CUDA graphs (FULL): 100%|██████████| 51/51 [00:04<00:00, 10.92it/s]
(EngineCore pid=33144) INFO 10-31 02:54:46 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.30 GiB
(EngineCore pid=33144) INFO 10-31 02:54:46 [gpu_worker.py:971] CUDA graph pool memory: 0.3 GiB (actual), 0.52 GiB (estimated), difference: 0.22 GiB (71.6%).
(EngineCore pid=33144) INFO 10-31 02:54:46 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 11.44 GiB for consumed memory (weights + non-torch), 2.12 GiB for peak activation, and 0.3 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=61540390810` (57.31 GiB) to fit into requested memory, or `--kv-cache-memory=69606456832` (64.83 GiB) to fully utilize gpu memory. Current kv cache memory in use is 57.76 GiB.
(EngineCore pid=33144) INFO 10-31 02:54:47 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=33144) INFO 10-31 02:54:48 [core.py:395] init engine (profile, create kv cache, warmup model) took 167.36 s (compilation: 23.99 s)
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 5.68it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 7.46it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 7.11it/s]
(EngineCore pid=33144) INFO 10-31 02:54:48 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=32715) WARNING 10-31 02:54:48 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=33144) WARNING 10-31 02:54:48 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=33144) WARNING 10-31 02:54:49 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=33144) WARNING 10-31 02:54:50 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
[rank0]:[W1031 02:55:13.175073068 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 383778816 bytes (free: 211681280, total: 85094825984).
[rank0]:[W1031 02:55:22.194705088 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 492830720 bytes (free: 408813568, total: 85094825984).
progress {"time": "02:55:48", "elapsed_min": 1.0, "submitted": 260, "finished": 4, "in_flight": 256, "gen_tokens": 6887, "tok_per_s": 115, "chain_tokens_median": 428.0, "chain_tokens_p90": 563.5, "chain_tokens_max": 601, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 125.8}
progress {"time": "02:56:48", "elapsed_min": 2.0, "submitted": 296, "finished": 40, "in_flight": 256, "gen_tokens": 101154, "tok_per_s": 843, "chain_tokens_median": 602.5, "chain_tokens_p90": 902.6999999999999, "chain_tokens_max": 1334, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 124.8}
progress {"time": "02:57:48", "elapsed_min": 3.0, "submitted": 324, "finished": 68, "in_flight": 256, "gen_tokens": 200050, "tok_per_s": 1111, "chain_tokens_median": 651.0, "chain_tokens_p90": 1132.1000000000001, "chain_tokens_max": 1958, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 123.8}
progress {"time": "02:58:48", "elapsed_min": 4.0, "submitted": 341, "finished": 85, "in_flight": 256, "gen_tokens": 269649, "tok_per_s": 1123, "chain_tokens_median": 685.5, "chain_tokens_p90": 1307.5000000000005, "chain_tokens_max": 2310, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 122.8}
progress {"time": "02:59:48", "elapsed_min": 5.0, "submitted": 358, "finished": 102, "in_flight": 256, "gen_tokens": 358592, "tok_per_s": 1195, "chain_tokens_median": 724.5, "chain_tokens_p90": 1513.3, "chain_tokens_max": 3095, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 121.8}
progress {"time": "03:00:48", "elapsed_min": 6.0, "submitted": 379, "finished": 123, "in_flight": 256, "gen_tokens": 498088, "tok_per_s": 1384, "chain_tokens_median": 774.0, "chain_tokens_p90": 1992.0, "chain_tokens_max": 3682, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 120.8}
progress {"time": "03:01:48", "elapsed_min": 7.0, "submitted": 397, "finished": 141, "in_flight": 256, "gen_tokens": 638033, "tok_per_s": 1519, "chain_tokens_median": 820.5, "chain_tokens_p90": 2423.2999999999997, "chain_tokens_max": 4179, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 119.8}
progress {"time": "03:02:48", "elapsed_min": 8.0, "submitted": 415, "finished": 159, "in_flight": 256, "gen_tokens": 785075, "tok_per_s": 1636, "chain_tokens_median": 859.0, "chain_tokens_p90": 2849.0, "chain_tokens_max": 4470, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 118.8}
progress {"time": "03:03:48", "elapsed_min": 9.0, "submitted": 438, "finished": 182, "in_flight": 256, "gen_tokens": 998263, "tok_per_s": 1849, "chain_tokens_median": 898.0, "chain_tokens_p90": 3189.2000000000016, "chain_tokens_max": 5422, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 117.8}
progress {"time": "03:04:48", "elapsed_min": 10.0, "submitted": 456, "finished": 200, "in_flight": 256, "gen_tokens": 1191397, "tok_per_s": 1986, "chain_tokens_median": 934.0, "chain_tokens_p90": 3527.6000000000004, "chain_tokens_max": 5720, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 116.8}
progress {"time": "03:05:48", "elapsed_min": 11.0, "submitted": 472, "finished": 216, "in_flight": 256, "gen_tokens": 1382639, "tok_per_s": 2095, "chain_tokens_median": 951.0, "chain_tokens_p90": 3925.200000000001, "chain_tokens_max": 6364, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 115.8}
progress {"time": "03:06:48", "elapsed_min": 12.0, "submitted": 489, "finished": 233, "in_flight": 256, "gen_tokens": 1580792, "tok_per_s": 2195, "chain_tokens_median": 995.5, "chain_tokens_p90": 4121.7, "chain_tokens_max": 7296, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 114.8}
progress {"time": "03:07:48", "elapsed_min": 13.0, "submitted": 505, "finished": 249, "in_flight": 256, "gen_tokens": 1822496, "tok_per_s": 2336, "chain_tokens_median": 1051.0, "chain_tokens_p90": 4395.0, "chain_tokens_max": 7713, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 113.8}
./run_all.sh: line 17: 8.67: syntax error: invalid arithmetic operator (error token is ".67")
./run_all.sh: line 19: END: unbound variable
{"time": "2026-10-31T03:08:29Z", "event": "skip", "stage": "prep", "hours_left": }
./run_all.sh: line 19: END: unbound variable
{"time": "2026-10-31T03:08:29Z", "event": "skip", "stage": "probe", "hours_left": }
./run_all.sh: line 19: END: unbound variable
{"time": "2026-10-31T03:08:29Z", "event": "skip", "stage": "baseline", "hours_left": }
./run_all.sh: line 41: END: unbound variable
RUN_ALL_EXIT 1
{"time": "2026-10-31T03:09:01Z", "event": "skip", "stage": "prep", "hours_left": 6.35}
{"time": "2026-10-31T03:09:01Z", "event": "skip", "stage": "probe", "hours_left": 6.35}
{"time": "2026-10-31T03:09:01Z", "event": "skip", "stage": "baseline", "hours_left": 6.35}
{"time": "2026-10-31T03:09:01Z", "event": "start", "stage": "generate", "hours_left": 6.35}
{"to_generate": 17571, "already_done": 380, "minutes_left": 152.5916556874911}
(MainProcess pid=34545) INFO 10-31 03:09:13 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration
(MainProcess pid=34545) INFO 10-31 03:09:13 [model.py:2152] Using max model len 32768
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1261.92it/s]
(MainProcess pid=34545) INFO 10-31 03:09:15 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=34545) INFO 10-31 03:09:15 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 7.07it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 12.43it/s]
(MainProcess pid=34545) INFO 10-31 03:09:16 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=34545) WARNING 10-31 03:09:27 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
(EngineCore pid=34971) INFO 10-31 03:09:38 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=34971) INFO 10-31 03:09:39 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_07bc7aa7b635477393202b7f37993a68 backend=nccl
(EngineCore pid=34971) INFO 10-31 03:09:39 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=34971) INFO 10-31 03:09:39 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=34971) INFO 10-31 03:09:41 [model_runner.py:392] Loading model from scratch...
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1120.55it/s]
(EngineCore pid=34971) INFO 10-31 03:09:42 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=34971) INFO 10-31 03:09:42 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=34971) INFO 10-31 03:09:42 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=34971) INFO 10-31 03:09:42 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=34971) INFO 10-31 03:09:42 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=34971) INFO 10-31 03:09:43 [base_loader.py:82] Initializing model took 2.141693 seconds
(EngineCore pid=34971) INFO 10-31 03:09:43 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.17 GiB.
(EngineCore pid=34971) INFO 10-31 03:09:43 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 1.73it/s]
(EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:43 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=34971) WARNING 10-31 03:09:44 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.85it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.83it/s]
(EngineCore pid=34971) INFO 10-31 03:09:44 [default_loader.py:501] Loading weights took 1.10 seconds
(EngineCore pid=34971) INFO 10-31 03:09:45 [model_runner.py:424] Model loading took 8.56 GiB memory and 5.110027 seconds
(EngineCore pid=34971) INFO 10-31 03:09:45 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=34971) INFO 10-31 03:09:45 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=34971) INFO 10-31 03:09:45 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=34971) INFO 10-31 03:09:45 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=34971) INFO 10-31 03:09:45 [utils.py:347] Using LBNHC KV cache layout.
[transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning.
(MainProcess pid=34545) INFO 10-31 03:09:48 [base.py:280] Multi-modal warmup completed in 20.651s
(EngineCore pid=34971) INFO 10-31 03:09:53 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=34971) INFO 10-31 03:09:57 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=13 num_submods=33
(EngineCore pid=34971) INFO 10-31 03:09:57 [decorators.py:319] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/cc0f0cae53dc3e1375a6d2ed4c88238808e3367e1d9abaa0419cc3d58ebc1b12/rank_0_0/model
(EngineCore pid=34971) INFO 10-31 03:09:57 [monitor.py:62] [backbone] torch.compile took 0.63 s in total
(EngineCore pid=34971) INFO 10-31 03:09:57 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.32 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:15, 2.21it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:11, 2.80it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:10, 3.12it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:09, 3.28it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:08, 3.41it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:01<00:08, 3.54it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:07, 3.63it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:07, 3.84it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:02<00:07, 3.66it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:02<00:06, 3.62it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:06, 3.57it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:06, 3.66it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:03<00:05, 3.78it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:03<00:05, 3.98it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:04, 4.36it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:03, 4.83it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:02, 6.95it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:04<00:01, 9.14it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:04<00:01, 10.97it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:04<00:00, 12.66it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:04<00:00, 14.15it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:04<00:00, 15.24it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:05<00:00, 15.93it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 16.68it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:05<00:00, 17.16it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:05<00:00, 6.55it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 23.31it/s]
(EngineCore pid=34971) INFO 10-31 03:10:05 [model_runner.py:1127] Graph capturing finished in 6 secs, took 0.28 GiB
(EngineCore pid=34971) INFO 10-31 03:10:05 [gpu_worker.py:759] Available KV cache memory: 60.55 GiB
(EngineCore pid=34971) INFO 10-31 03:10:05 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=34971) INFO 10-31 03:10:05 [kv_cache_utils.py:2327] GPU KV cache size: 1,784,193 tokens, Maximum concurrency for 32,768 tokens per request: 54.45x
(EngineCore pid=34971) INFO 10-31 03:10:05 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=34971) INFO 10-31 03:10:05 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=34971) INFO 10-31 03:10:05 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=34971) INFO 10-31 03:10:05 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=34971) INFO 10-31 03:10:05 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=34971) INFO 10-31 03:10:05 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)].
(EngineCore pid=34971) INFO 10-31 03:10:05 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:14, 2.31it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.57it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:12, 2.61it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.63it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:10, 2.76it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:10, 2.88it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:08, 3.12it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:08, 3.31it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:07, 3.26it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:07, 3.30it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:06, 3.47it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:06, 3.76it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:05, 4.04it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:04, 4.32it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:04, 4.71it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:03, 4.95it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:02, 6.92it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:04<00:01, 8.71it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:04<00:01, 10.20it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:05<00:00, 11.14it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:05<00:00, 12.18it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:05<00:00, 12.66it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:05<00:00, 13.23it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 13.83it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:05<00:00, 13.97it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:05<00:00, 5.96it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 21.37it/s]Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 20.55it/s]Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 21.18it/s]Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 20.86it/s]Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 21.40it/s]Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 21.12it/s]Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 21.15it/s]
(EngineCore pid=34971) INFO 10-31 03:10:15 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.14 GiB
(EngineCore pid=34971) INFO 10-31 03:10:15 [gpu_worker.py:971] CUDA graph pool memory: 0.14 GiB (actual), 0.32 GiB (estimated), difference: 0.18 GiB (132.9%).
(EngineCore pid=34971) INFO 10-31 03:10:15 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 8.86 GiB for consumed memory (weights + non-torch), 1.92 GiB for peak activation, and 0.14 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=64709187482` (60.27 GiB) to fit into requested memory, or `--kv-cache-memory=72775253504` (67.78 GiB) to fully utilize gpu memory. Current kv cache memory in use is 60.55 GiB.
(EngineCore pid=34971) INFO 10-31 03:10:16 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=34971) INFO 10-31 03:10:16 [core.py:395] init engine (profile, create kv cache, warmup model) took 30.91 s (compilation: 0.63 s)
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 2.27it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 2.46it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 2.43it/s]
(EngineCore pid=34971) INFO 10-31 03:10:18 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=34545) WARNING 10-31 03:10:18 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=34971) WARNING 10-31 03:10:18 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=34971) WARNING 10-31 03:10:18 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=34971) WARNING 10-31 03:10:18 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "03:11:16", "elapsed_min": 1.0, "submitted": 257, "finished": 1, "in_flight": 256, "gen_tokens": 2479, "tok_per_s": 41, "chain_tokens_median": 611.0, "chain_tokens_p90": 688.8, "chain_tokens_max": 708, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 150.5}
(MainProcess pid=34545) INFO 10-31 03:11:16 [loggers.py:335] Engine 000: Avg prompt throughput: 893.0 tokens/s, Avg generation throughput: 5140.7 tokens/s, Running: 128 reqs, Waiting: 887 reqs, GPU KV cache usage: 29.2%, Prefix cache hit rate: 51.6%
progress {"time": "03:12:16", "elapsed_min": 2.0, "submitted": 260, "finished": 4, "in_flight": 256, "gen_tokens": 39184, "tok_per_s": 327, "chain_tokens_median": 2926.0, "chain_tokens_p90": 3548.5, "chain_tokens_max": 4055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 149.5}
(MainProcess pid=34545) INFO 10-31 03:12:16 [loggers.py:335] Engine 000: Avg prompt throughput: 105.7 tokens/s, Avg generation throughput: 4575.3 tokens/s, Running: 128 reqs, Waiting: 876 reqs, GPU KV cache usage: 39.5%, Prefix cache hit rate: 52.0%
progress {"time": "03:13:16", "elapsed_min": 3.0, "submitted": 265, "finished": 9, "in_flight": 256, "gen_tokens": 124509, "tok_per_s": 692, "chain_tokens_median": 3425.0, "chain_tokens_p90": 5575.0, "chain_tokens_max": 6435, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 148.5}
(MainProcess pid=34545) INFO 10-31 03:13:16 [loggers.py:335] Engine 000: Avg prompt throughput: 244.5 tokens/s, Avg generation throughput: 4180.3 tokens/s, Running: 128 reqs, Waiting: 848 reqs, GPU KV cache usage: 40.5%, Prefix cache hit rate: 51.0%
progress {"time": "03:14:16", "elapsed_min": 4.0, "submitted": 274, "finished": 18, "in_flight": 256, "gen_tokens": 324945, "tok_per_s": 1354, "chain_tokens_median": 4679.5, "chain_tokens_p90": 7200.1, "chain_tokens_max": 8208, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 147.5}
(MainProcess pid=34545) INFO 10-31 03:14:16 [loggers.py:335] Engine 000: Avg prompt throughput: 299.0 tokens/s, Avg generation throughput: 4190.9 tokens/s, Running: 128 reqs, Waiting: 843 reqs, GPU KV cache usage: 39.4%, Prefix cache hit rate: 53.4%
progress {"time": "03:15:16", "elapsed_min": 5.0, "submitted": 285, "finished": 29, "in_flight": 256, "gen_tokens": 574372, "tok_per_s": 1914, "chain_tokens_median": 5331.0, "chain_tokens_p90": 7960.0, "chain_tokens_max": 10149, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 146.5}
(MainProcess pid=34545) INFO 10-31 03:15:16 [loggers.py:335] Engine 000: Avg prompt throughput: 227.8 tokens/s, Avg generation throughput: 4125.1 tokens/s, Running: 128 reqs, Waiting: 853 reqs, GPU KV cache usage: 42.7%, Prefix cache hit rate: 53.5%
{"time": "2026-10-31T03:16:02Z", "event": "skip", "stage": "prep", "hours_left": 6.23}
{"time": "2026-10-31T03:16:02Z", "event": "skip", "stage": "probe", "hours_left": 6.23}
{"time": "2026-10-31T03:16:02Z", "event": "skip", "stage": "baseline", "hours_left": 6.23}
{"time": "2026-10-31T03:16:02Z", "event": "start", "stage": "generate", "hours_left": 6.23}
{"to_generate": 17539, "already_done": 412, "minutes_left": 145.53148780266443}
(MainProcess pid=35900) INFO 10-31 03:16:18 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration
(MainProcess pid=35900) INFO 10-31 03:16:18 [model.py:2152] Using max model len 32768
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1383.84it/s]
(MainProcess pid=35900) INFO 10-31 03:16:20 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=35900) INFO 10-31 03:16:20 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 2.41it/s]Parse safetensors files: 100%|██████████| 2/2 [00:01<00:00, 1.40it/s]Parse safetensors files: 100%|██████████| 2/2 [00:01<00:00, 1.50it/s]
(MainProcess pid=35900) INFO 10-31 03:16:21 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=35900) WARNING 10-31 03:16:33 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
(EngineCore pid=36321) INFO 10-31 03:16:46 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=36321) INFO 10-31 03:16:48 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_2f31013152a343f79b3bc765d85c6842 backend=nccl
(EngineCore pid=36321) INFO 10-31 03:16:48 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=36321) INFO 10-31 03:16:48 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=36321) INFO 10-31 03:16:49 [model_runner.py:392] Loading model from scratch...
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1122.25it/s]
(EngineCore pid=36321) INFO 10-31 03:16:51 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=36321) INFO 10-31 03:16:51 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=36321) INFO 10-31 03:16:52 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=36321) INFO 10-31 03:16:52 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=36321) INFO 10-31 03:16:52 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=36321) INFO 10-31 03:16:53 [base_loader.py:82] Initializing model took 3.283994 seconds
(EngineCore pid=36321) INFO 10-31 03:16:53 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.42 GiB.
(EngineCore pid=36321) INFO 10-31 03:16:53 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 1.95it/s]
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:53 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=36321) WARNING 10-31 03:16:54 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.04it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.03it/s]
(EngineCore pid=36321) INFO 10-31 03:16:54 [default_loader.py:501] Loading weights took 0.99 seconds
[transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning.
(MainProcess pid=35900) INFO 10-31 03:16:55 [base.py:280] Multi-modal warmup completed in 21.908s
(EngineCore pid=36321) INFO 10-31 03:16:55 [model_runner.py:424] Model loading took 8.56 GiB memory and 6.200593 seconds
(EngineCore pid=36321) INFO 10-31 03:16:55 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=36321) INFO 10-31 03:16:55 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=36321) INFO 10-31 03:16:55 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=36321) INFO 10-31 03:16:55 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=36321) INFO 10-31 03:16:55 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=36321) INFO 10-31 03:17:03 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=36321) INFO 10-31 03:17:06 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=36321) INFO 10-31 03:17:11 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/0c75a73c22/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=36321) INFO 10-31 03:17:11 [backends.py:1166] [backbone] Dynamo bytecode transform time: 4.78 s
(EngineCore pid=36321) INFO 10-31 03:17:29 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 17.76 s
(EngineCore pid=36321) INFO 10-31 03:17:32 [backends.py:927] collected artifacts: 33 entries, 13 artifacts, 18863679 bytes total
(EngineCore pid=36321) INFO 10-31 03:17:32 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/280d235bd2883e7a75f93a251c5ae942da381615d0d2c83834f39fb3bfac00c9/rank_0_0/model
(EngineCore pid=36321) INFO 10-31 03:17:32 [monitor.py:62] [backbone] torch.compile took 25.51 s in total
(EngineCore pid=36321) INFO 10-31 03:18:59 [monitor.py:91] [backbone] Initial profiling/warmup run took 86.76 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:49, 1.00it/s]Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:42, 1.14it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:31, 1.50it/s]Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:02<00:25, 1.87it/s]Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:02<00:21, 2.12it/s]Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:03<00:22, 1.97it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:23, 1.90it/s]Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:04<00:22, 1.94it/s]Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:04<00:20, 2.03it/s]Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:05<00:19, 2.08it/s]Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:05<00:16, 2.39it/s]Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:05<00:14, 2.66it/s]Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:06<00:13, 2.79it/s]Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:06<00:12, 2.91it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:06<00:11, 3.18it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:07<00:10, 3.45it/s]Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:08<00:17, 1.90it/s]Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:14, 2.29it/s]Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:08<00:08, 3.80it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:08<00:05, 5.53it/s]Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:08<00:03, 7.35it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:08<00:02, 9.06it/s]Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:02, 10.76it/s]Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:09<00:01, 12.23it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:09<00:01, 13.48it/s]Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:09<00:01, 13.97it/s]Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:09<00:01, 14.85it/s]Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:09<00:00, 15.29it/s]Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:09<00:00, 15.65it/s]Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:09<00:00, 16.22it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:09<00:00, 16.24it/s]Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:10<00:00, 16.68it/s]Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:10<00:00, 17.00it/s]Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:10<00:00, 17.21it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 4.84it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 21.04it/s]
(EngineCore pid=36321) INFO 10-31 03:19:13 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.40 GiB
(EngineCore pid=36321) INFO 10-31 03:19:14 [gpu_worker.py:759] Available KV cache memory: 58.06 GiB
(EngineCore pid=36321) INFO 10-31 03:19:14 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8940 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9060. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=36321) INFO 10-31 03:19:14 [kv_cache_utils.py:2327] GPU KV cache size: 1,711,059 tokens, Maximum concurrency for 32,768 tokens per request: 52.22x
(EngineCore pid=36321) INFO 10-31 03:19:14 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=36321) INFO 10-31 03:19:14 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=36321) INFO 10-31 03:19:14 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=36321) INFO 10-31 03:19:14 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=36321) INFO 10-31 03:19:14 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=36321) INFO 10-31 03:19:16 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)].
(EngineCore pid=36321) INFO 10-31 03:19:16 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:41, 1.20it/s]Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:41, 1.19it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:39, 1.22it/s]Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:36, 1.30it/s]Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:33, 1.36it/s]Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:30, 1.46it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:28, 1.56it/s]Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:25, 1.69it/s]Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:05<00:22, 1.83it/s]Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:20, 2.00it/s]Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:17, 2.24it/s]Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:16, 2.43it/s]Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:14, 2.61it/s]Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:13, 2.83it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:12, 2.94it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:08<00:10, 3.48it/s]Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:06, 5.42it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:08<00:05, 5.74it/s]Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:08<00:03, 7.76it/s]Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:08<00:02, 9.52it/s]Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:08<00:02, 9.73it/s]Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:08<00:02, 10.98it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:09<00:01, 12.19it/s]Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:09<00:01, 13.14it/s]Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:09<00:01, 12.30it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:09<00:01, 13.19it/s]Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:09<00:01, 13.88it/s]Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:09<00:00, 14.37it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:09<00:00, 13.15it/s]Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:10<00:00, 14.00it/s]Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:10<00:00, 14.54it/s]Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:10<00:00, 12.50it/s]Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:10<00:00, 13.36it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 14.13it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 4.81it/s]
Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 24.32it/s]Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 18.85it/s]Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 20.69it/s]Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:01, 21.66it/s]Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:01, 19.61it/s]Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 20.73it/s]Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 21.61it/s]Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 21.02it/s]Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 20.80it/s]Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 21.87it/s]Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 22.54it/s]Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 21.22it/s]
(EngineCore pid=36321) INFO 10-31 03:19:48 [model_runner.py:1127] Graph capturing finished in 13 secs, took 0.24 GiB
(EngineCore pid=36321) INFO 10-31 03:19:48 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (100.8%).
(EngineCore pid=36321) INFO 10-31 03:19:48 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 11.19 GiB for consumed memory (weights + non-torch), 2.08 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=61932558234` (57.68 GiB) to fit into requested memory, or `--kv-cache-memory=69998624256` (65.19 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.06 GiB.
(EngineCore pid=36321) INFO 10-31 03:19:49 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=36321) INFO 10-31 03:19:49 [core.py:395] init engine (profile, create kv cache, warmup model) took 173.89 s (compilation: 25.51 s)
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 6.70it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 12.37it/s]
(EngineCore pid=36321) INFO 10-31 03:19:50 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=35900) WARNING 10-31 03:19:50 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=36321) WARNING 10-31 03:19:51 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=36321) WARNING 10-31 03:19:51 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=36321) WARNING 10-31 03:19:53 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "03:20:49", "elapsed_min": 1.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": 141.0}
(MainProcess pid=35900) INFO 10-31 03:20:49 [loggers.py:335] Engine 000: Avg prompt throughput: 1527.8 tokens/s, Avg generation throughput: 4786.9 tokens/s, Running: 256 reqs, Waiting: 759 reqs, GPU KV cache usage: 44.9%, Prefix cache hit rate: 49.6%
progress {"time": "03:21:49", "elapsed_min": 2.0, "submitted": 259, "finished": 3, "in_flight": 256, "gen_tokens": 13116, "tok_per_s": 109, "chain_tokens_median": 911.0, "chain_tokens_p90": 1659.3000000000002, "chain_tokens_max": 1913, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 140.0}
(MainProcess pid=35900) INFO 10-31 03:21:49 [loggers.py:335] Engine 000: Avg prompt throughput: 186.1 tokens/s, Avg generation throughput: 5997.6 tokens/s, Running: 256 reqs, Waiting: 737 reqs, GPU KV cache usage: 60.9%, Prefix cache hit rate: 48.8%
progress {"time": "03:22:49", "elapsed_min": 3.0, "submitted": 269, "finished": 13, "in_flight": 256, "gen_tokens": 67831, "tok_per_s": 377, "chain_tokens_median": 911.0, "chain_tokens_p90": 2925.899999999999, "chain_tokens_max": 3527, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 139.0}
(MainProcess pid=35900) INFO 10-31 03:22:49 [loggers.py:335] Engine 000: Avg prompt throughput: 300.2 tokens/s, Avg generation throughput: 5148.4 tokens/s, Running: 256 reqs, Waiting: 719 reqs, GPU KV cache usage: 70.5%, Prefix cache hit rate: 48.5%
progress {"time": "03:23:49", "elapsed_min": 4.0, "submitted": 284, "finished": 28, "in_flight": 256, "gen_tokens": 212170, "tok_per_s": 884, "chain_tokens_median": 1286.5, "chain_tokens_p90": 4229.0, "chain_tokens_max": 4835, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 138.0}
(MainProcess pid=35900) INFO 10-31 03:23:49 [loggers.py:335] Engine 000: Avg prompt throughput: 476.9 tokens/s, Avg generation throughput: 4717.0 tokens/s, Running: 256 reqs, Waiting: 691 reqs, GPU KV cache usage: 71.0%, Prefix cache hit rate: 47.7%
progress {"time": "03:24:49", "elapsed_min": 5.0, "submitted": 306, "finished": 50, "in_flight": 256, "gen_tokens": 419649, "tok_per_s": 1399, "chain_tokens_median": 1344.0, "chain_tokens_p90": 4643.1, "chain_tokens_max": 6017, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 137.0}
(MainProcess pid=35900) INFO 10-31 03:24:49 [loggers.py:335] Engine 000: Avg prompt throughput: 486.1 tokens/s, Avg generation throughput: 4686.8 tokens/s, Running: 255 reqs, Waiting: 672 reqs, GPU KV cache usage: 70.5%, Prefix cache hit rate: 46.0%
progress {"time": "03:25:49", "elapsed_min": 6.0, "submitted": 329, "finished": 73, "in_flight": 256, "gen_tokens": 692970, "tok_per_s": 1925, "chain_tokens_median": 1565.5, "chain_tokens_p90": 5095.800000000001, "chain_tokens_max": 7096, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 136.0}
(MainProcess pid=35900) INFO 10-31 03:25:49 [loggers.py:335] Engine 000: Avg prompt throughput: 521.3 tokens/s, Avg generation throughput: 4691.0 tokens/s, Running: 256 reqs, Waiting: 662 reqs, GPU KV cache usage: 69.3%, Prefix cache hit rate: 45.1%
progress {"time": "03:26:49", "elapsed_min": 7.0, "submitted": 355, "finished": 99, "in_flight": 256, "gen_tokens": 965101, "tok_per_s": 2298, "chain_tokens_median": 1561.5, "chain_tokens_p90": 5249.0, "chain_tokens_max": 7945, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 135.0}
(MainProcess pid=35900) INFO 10-31 03:26:49 [loggers.py:335] Engine 000: Avg prompt throughput: 419.9 tokens/s, Avg generation throughput: 4759.5 tokens/s, Running: 256 reqs, Waiting: 669 reqs, GPU KV cache usage: 69.1%, Prefix cache hit rate: 44.8%
progress {"time": "03:27:49", "elapsed_min": 8.0, "submitted": 370, "finished": 114, "in_flight": 256, "gen_tokens": 1188153, "tok_per_s": 2475, "chain_tokens_median": 1668.0, "chain_tokens_p90": 5879.0, "chain_tokens_max": 9132, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 134.0}
(MainProcess pid=35900) INFO 10-31 03:27:49 [loggers.py:335] Engine 000: Avg prompt throughput: 452.1 tokens/s, Avg generation throughput: 4806.6 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 70.0%, Prefix cache hit rate: 44.2%
progress {"time": "03:28:49", "elapsed_min": 9.0, "submitted": 401, "finished": 145, "in_flight": 256, "gen_tokens": 1522170, "tok_per_s": 2819, "chain_tokens_median": 1613.0, "chain_tokens_p90": 6142.000000000003, "chain_tokens_max": 10224, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 133.0}
(MainProcess pid=35900) INFO 10-31 03:28:49 [loggers.py:335] Engine 000: Avg prompt throughput: 411.4 tokens/s, Avg generation throughput: 4811.0 tokens/s, Running: 256 reqs, Waiting: 681 reqs, GPU KV cache usage: 72.4%, Prefix cache hit rate: 44.3%
progress {"time": "03:29:49", "elapsed_min": 10.0, "submitted": 423, "finished": 167, "in_flight": 256, "gen_tokens": 1732163, "tok_per_s": 2887, "chain_tokens_median": 1548.5, "chain_tokens_p90": 6075.700000000005, "chain_tokens_max": 11527, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 132.0}
(MainProcess pid=35900) INFO 10-31 03:29:49 [loggers.py:335] Engine 000: Avg prompt throughput: 499.2 tokens/s, Avg generation throughput: 4588.7 tokens/s, Running: 255 reqs, Waiting: 671 reqs, GPU KV cache usage: 72.1%, Prefix cache hit rate: 44.0%
progress {"time": "03:30:49", "elapsed_min": 11.0, "submitted": 446, "finished": 190, "in_flight": 256, "gen_tokens": 2012478, "tok_per_s": 3049, "chain_tokens_median": 1573.5, "chain_tokens_p90": 6281.8, "chain_tokens_max": 12596, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 131.0}
(MainProcess pid=35900) INFO 10-31 03:30:49 [loggers.py:335] Engine 000: Avg prompt throughput: 471.7 tokens/s, Avg generation throughput: 4686.9 tokens/s, Running: 255 reqs, Waiting: 668 reqs, GPU KV cache usage: 71.3%, Prefix cache hit rate: 41.5%
progress {"time": "03:31:49", "elapsed_min": 12.0, "submitted": 467, "finished": 211, "in_flight": 256, "gen_tokens": 2263855, "tok_per_s": 3144, "chain_tokens_median": 1560.0, "chain_tokens_p90": 6360.4000000000015, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 130.0}
(MainProcess pid=35900) INFO 10-31 03:31:49 [loggers.py:335] Engine 000: Avg prompt throughput: 483.3 tokens/s, Avg generation throughput: 4652.8 tokens/s, Running: 256 reqs, Waiting: 669 reqs, GPU KV cache usage: 73.7%, Prefix cache hit rate: 40.1%
progress {"time": "03:32:49", "elapsed_min": 13.0, "submitted": 482, "finished": 226, "in_flight": 256, "gen_tokens": 2413935, "tok_per_s": 3095, "chain_tokens_median": 1565.5, "chain_tokens_p90": 6310.000000000002, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 129.0}
(MainProcess pid=35900) INFO 10-31 03:32:49 [loggers.py:335] Engine 000: Avg prompt throughput: 526.6 tokens/s, Avg generation throughput: 4520.7 tokens/s, Running: 256 reqs, Waiting: 649 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 41.9%
progress {"time": "03:33:49", "elapsed_min": 14.0, "submitted": 505, "finished": 249, "in_flight": 256, "gen_tokens": 2738168, "tok_per_s": 3259, "chain_tokens_median": 1643.0, "chain_tokens_p90": 6368.5, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 128.0}
(MainProcess pid=35900) INFO 10-31 03:33:49 [loggers.py:335] Engine 000: Avg prompt throughput: 409.8 tokens/s, Avg generation throughput: 4486.6 tokens/s, Running: 256 reqs, Waiting: 658 reqs, GPU KV cache usage: 77.4%, Prefix cache hit rate: 41.2%
progress {"time": "03:34:49", "elapsed_min": 15.0, "submitted": 519, "finished": 263, "in_flight": 256, "gen_tokens": 2855036, "tok_per_s": 3172, "chain_tokens_median": 1615.5, "chain_tokens_p90": 6364.799999999999, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 127.0}
(MainProcess pid=35900) INFO 10-31 03:34:49 [loggers.py:335] Engine 000: Avg prompt throughput: 374.6 tokens/s, Avg generation throughput: 4392.8 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 79.3%, Prefix cache hit rate: 40.4%
progress {"time": "03:35:49", "elapsed_min": 16.0, "submitted": 537, "finished": 281, "in_flight": 256, "gen_tokens": 3076344, "tok_per_s": 3204, "chain_tokens_median": 1685.5, "chain_tokens_p90": 6360.4000000000015, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 126.0}
(MainProcess pid=35900) INFO 10-31 03:35:49 [loggers.py:335] Engine 000: Avg prompt throughput: 410.6 tokens/s, Avg generation throughput: 4410.1 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 82.4%, Prefix cache hit rate: 41.6%
progress {"time": "03:36:49", "elapsed_min": 17.0, "submitted": 554, "finished": 298, "in_flight": 256, "gen_tokens": 3290962, "tok_per_s": 3226, "chain_tokens_median": 1711.0, "chain_tokens_p90": 6386.200000000002, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 125.0}
(MainProcess pid=35900) INFO 10-31 03:36:49 [loggers.py:335] Engine 000: Avg prompt throughput: 289.9 tokens/s, Avg generation throughput: 4346.3 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 41.5%
progress {"time": "03:37:49", "elapsed_min": 18.0, "submitted": 565, "finished": 309, "in_flight": 256, "gen_tokens": 3386051, "tok_per_s": 3135, "chain_tokens_median": 1685.5, "chain_tokens_p90": 6379.0, "chain_tokens_max": 13039, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 124.0}
(MainProcess pid=35900) INFO 10-31 03:37:49 [loggers.py:335] Engine 000: Avg prompt throughput: 362.4 tokens/s, Avg generation throughput: 4196.7 tokens/s, Running: 256 reqs, Waiting: 618 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 42.0%
progress {"time": "03:38:49", "elapsed_min": 19.0, "submitted": 585, "finished": 329, "in_flight": 256, "gen_tokens": 3747837, "tok_per_s": 3287, "chain_tokens_median": 1776.5, "chain_tokens_p90": 6518.5, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 123.0}
(MainProcess pid=35900) INFO 10-31 03:38:49 [loggers.py:335] Engine 000: Avg prompt throughput: 336.5 tokens/s, Avg generation throughput: 4124.2 tokens/s, Running: 256 reqs, Waiting: 626 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 42.3%
progress {"time": "03:39:49", "elapsed_min": 20.0, "submitted": 606, "finished": 350, "in_flight": 256, "gen_tokens": 4070930, "tok_per_s": 3392, "chain_tokens_median": 1818.0, "chain_tokens_p90": 6618.1, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 122.0}
(MainProcess pid=35900) INFO 10-31 03:39:49 [loggers.py:335] Engine 000: Avg prompt throughput: 519.7 tokens/s, Avg generation throughput: 4038.5 tokens/s, Running: 256 reqs, Waiting: 614 reqs, GPU KV cache usage: 80.7%, Prefix cache hit rate: 41.5%
progress {"time": "03:40:49", "elapsed_min": 21.0, "submitted": 631, "finished": 375, "in_flight": 256, "gen_tokens": 4320153, "tok_per_s": 3428, "chain_tokens_median": 1799.5, "chain_tokens_p90": 6514.9000000000015, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 121.0}
(MainProcess pid=35900) INFO 10-31 03:40:49 [loggers.py:335] Engine 000: Avg prompt throughput: 412.0 tokens/s, Avg generation throughput: 4128.0 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 81.4%, Prefix cache hit rate: 41.3%
progress {"time": "03:41:49", "elapsed_min": 22.0, "submitted": 655, "finished": 399, "in_flight": 256, "gen_tokens": 4543207, "tok_per_s": 3442, "chain_tokens_median": 1776.5, "chain_tokens_p90": 6470.0, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 120.0}
(MainProcess pid=35900) INFO 10-31 03:41:49 [loggers.py:335] Engine 000: Avg prompt throughput: 495.3 tokens/s, Avg generation throughput: 4080.9 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 82.1%, Prefix cache hit rate: 42.7%
progress {"time": "03:42:49", "elapsed_min": 23.0, "submitted": 670, "finished": 414, "in_flight": 256, "gen_tokens": 4735792, "tok_per_s": 3431, "chain_tokens_median": 1775.0, "chain_tokens_p90": 6518.5, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 119.0}
(MainProcess pid=35900) INFO 10-31 03:42:49 [loggers.py:335] Engine 000: Avg prompt throughput: 405.4 tokens/s, Avg generation throughput: 4141.2 tokens/s, Running: 255 reqs, Waiting: 621 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 43.4%
progress {"time": "03:43:49", "elapsed_min": 24.0, "submitted": 687, "finished": 431, "in_flight": 256, "gen_tokens": 4932832, "tok_per_s": 3425, "chain_tokens_median": 1791.5, "chain_tokens_p90": 6509.500000000001, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 118.0}
(MainProcess pid=35900) INFO 10-31 03:43:49 [loggers.py:335] Engine 000: Avg prompt throughput: 376.4 tokens/s, Avg generation throughput: 4038.6 tokens/s, Running: 255 reqs, Waiting: 615 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 42.6%
progress {"time": "03:44:49", "elapsed_min": 25.0, "submitted": 706, "finished": 450, "in_flight": 256, "gen_tokens": 5123416, "tok_per_s": 3415, "chain_tokens_median": 1775.0, "chain_tokens_p90": 6489.0, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 117.0}
(MainProcess pid=35900) INFO 10-31 03:44:49 [loggers.py:335] Engine 000: Avg prompt throughput: 463.0 tokens/s, Avg generation throughput: 3885.2 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 89.9%, Prefix cache hit rate: 43.1%
progress {"time": "03:45:49", "elapsed_min": 26.0, "submitted": 726, "finished": 470, "in_flight": 256, "gen_tokens": 5392241, "tok_per_s": 3456, "chain_tokens_median": 1791.5, "chain_tokens_p90": 6500.500000000002, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 116.0}
(MainProcess pid=35900) INFO 10-31 03:45:49 [loggers.py:335] Engine 000: Avg prompt throughput: 582.9 tokens/s, Avg generation throughput: 3799.7 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 89.4%, Prefix cache hit rate: 42.3%
progress {"time": "03:46:49", "elapsed_min": 27.0, "submitted": 752, "finished": 496, "in_flight": 256, "gen_tokens": 5643489, "tok_per_s": 3483, "chain_tokens_median": 1783.0, "chain_tokens_p90": 6453.4, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 115.0}
(MainProcess pid=35900) INFO 10-31 03:46:49 [loggers.py:335] Engine 000: Avg prompt throughput: 443.7 tokens/s, Avg generation throughput: 3778.2 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 88.1%, Prefix cache hit rate: 43.1%
progress {"time": "03:47:49", "elapsed_min": 28.0, "submitted": 773, "finished": 517, "in_flight": 256, "gen_tokens": 5881957, "tok_per_s": 3501, "chain_tokens_median": 1791.5, "chain_tokens_p90": 6442.3, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 114.0}
(MainProcess pid=35900) INFO 10-31 03:47:49 [loggers.py:335] Engine 000: Avg prompt throughput: 367.3 tokens/s, Avg generation throughput: 3915.1 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 90.6%, Prefix cache hit rate: 43.1%
progress {"time": "03:48:49", "elapsed_min": 29.0, "submitted": 787, "finished": 531, "in_flight": 256, "gen_tokens": 6035067, "tok_per_s": 3468, "chain_tokens_median": 1808.0, "chain_tokens_p90": 6436.4, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 113.0}
(MainProcess pid=35900) INFO 10-31 03:48:49 [loggers.py:335] Engine 000: Avg prompt throughput: 333.0 tokens/s, Avg generation throughput: 3855.8 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 95.2%, Prefix cache hit rate: 43.9%
progress {"time": "03:49:49", "elapsed_min": 30.0, "submitted": 800, "finished": 544, "in_flight": 256, "gen_tokens": 6123273, "tok_per_s": 3402, "chain_tokens_median": 1764.5, "chain_tokens_p90": 6416.0, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 112.0}
(MainProcess pid=35900) INFO 10-31 03:49:49 [loggers.py:335] Engine 000: Avg prompt throughput: 308.7 tokens/s, Avg generation throughput: 3666.5 tokens/s, Running: 251 reqs, Waiting: 648 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 43.1%
progress {"time": "03:50:49", "elapsed_min": 31.0, "submitted": 810, "finished": 554, "in_flight": 256, "gen_tokens": 6272934, "tok_per_s": 3372, "chain_tokens_median": 1776.0, "chain_tokens_p90": 6436.0, "chain_tokens_max": 20011, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 111.0}
(MainProcess pid=35900) INFO 10-31 03:50:49 [loggers.py:335] Engine 000: Avg prompt throughput: 126.1 tokens/s, Avg generation throughput: 2651.1 tokens/s, Running: 238 reqs, Waiting: 661 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 42.3%
progress {"time": "03:51:49", "elapsed_min": 32.0, "submitted": 822, "finished": 566, "in_flight": 256, "gen_tokens": 6618372, "tok_per_s": 3447, "chain_tokens_median": 1801.5, "chain_tokens_p90": 6442.7, "chain_tokens_max": 32449, "parse_rate": 0.9969081272084805, "looped": 7, "minutes_to_deadline": 110.0}
(MainProcess pid=35900) INFO 10-31 03:51:49 [loggers.py:335] Engine 000: Avg prompt throughput: 341.7 tokens/s, Avg generation throughput: 3195.3 tokens/s, Running: 256 reqs, Waiting: 643 reqs, GPU KV cache usage: 94.6%, Prefix cache hit rate: 43.3%
progress {"time": "03:52:49", "elapsed_min": 33.0, "submitted": 837, "finished": 581, "in_flight": 256, "gen_tokens": 6988842, "tok_per_s": 3529, "chain_tokens_median": 1841.5, "chain_tokens_p90": 6492.700000000001, "chain_tokens_max": 32583, "parse_rate": 0.9948364888123924, "looped": 12, "minutes_to_deadline": 109.0}
(MainProcess pid=35900) INFO 10-31 03:52:49 [loggers.py:335] Engine 000: Avg prompt throughput: 322.2 tokens/s, Avg generation throughput: 3902.3 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 42.3%
progress {"time": "03:53:49", "elapsed_min": 34.0, "submitted": 854, "finished": 598, "in_flight": 256, "gen_tokens": 7192441, "tok_per_s": 3525, "chain_tokens_median": 1826.5, "chain_tokens_p90": 6509.800000000001, "chain_tokens_max": 32583, "parse_rate": 0.9949832775919732, "looped": 12, "minutes_to_deadline": 108.0}
(MainProcess pid=35900) INFO 10-31 03:53:49 [loggers.py:335] Engine 000: Avg prompt throughput: 418.1 tokens/s, Avg generation throughput: 3868.1 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 86.7%, Prefix cache hit rate: 44.1%
progress {"time": "03:54:49", "elapsed_min": 35.0, "submitted": 873, "finished": 617, "in_flight": 256, "gen_tokens": 7422706, "tok_per_s": 3534, "chain_tokens_median": 1836.5, "chain_tokens_p90": 6512.6, "chain_tokens_max": 32583, "parse_rate": 0.9951377633711507, "looped": 12, "minutes_to_deadline": 107.0}
(MainProcess pid=35900) INFO 10-31 03:54:49 [loggers.py:335] Engine 000: Avg prompt throughput: 363.3 tokens/s, Avg generation throughput: 3884.6 tokens/s, Running: 256 reqs, Waiting: 627 reqs, GPU KV cache usage: 88.1%, Prefix cache hit rate: 44.3%
progress {"time": "03:55:49", "elapsed_min": 36.0, "submitted": 890, "finished": 634, "in_flight": 256, "gen_tokens": 7615307, "tok_per_s": 3525, "chain_tokens_median": 1856.5, "chain_tokens_p90": 6492.5, "chain_tokens_max": 32583, "parse_rate": 0.9952681388012619, "looped": 12, "minutes_to_deadline": 106.0}
(MainProcess pid=35900) INFO 10-31 03:55:49 [loggers.py:335] Engine 000: Avg prompt throughput: 286.0 tokens/s, Avg generation throughput: 3881.2 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 89.1%, Prefix cache hit rate: 43.0%
progress {"time": "03:56:49", "elapsed_min": 37.0, "submitted": 910, "finished": 654, "in_flight": 256, "gen_tokens": 7899181, "tok_per_s": 3558, "chain_tokens_median": 1856.5, "chain_tokens_p90": 6528.5, "chain_tokens_max": 32583, "parse_rate": 0.9950305810397554, "looped": 13, "minutes_to_deadline": 105.0}
(MainProcess pid=35900) INFO 10-31 03:56:49 [loggers.py:335] Engine 000: Avg prompt throughput: 463.4 tokens/s, Avg generation throughput: 3846.6 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 42.8%
progress {"time": "03:57:49", "elapsed_min": 38.0, "submitted": 930, "finished": 674, "in_flight": 256, "gen_tokens": 8145011, "tok_per_s": 3572, "chain_tokens_median": 1879.5, "chain_tokens_p90": 6528.5, "chain_tokens_max": 32583, "parse_rate": 0.9951780415430267, "looped": 13, "minutes_to_deadline": 104.0}
(MainProcess pid=35900) INFO 10-31 03:57:49 [loggers.py:335] Engine 000: Avg prompt throughput: 415.6 tokens/s, Avg generation throughput: 3893.4 tokens/s, Running: 255 reqs, Waiting: 626 reqs, GPU KV cache usage: 87.8%, Prefix cache hit rate: 41.6%
progress {"time": "03:58:49", "elapsed_min": 39.0, "submitted": 947, "finished": 691, "in_flight": 256, "gen_tokens": 8306419, "tok_per_s": 3549, "chain_tokens_median": 1852.5, "chain_tokens_p90": 6511.700000000001, "chain_tokens_max": 32583, "parse_rate": 0.9952966714905933, "looped": 13, "minutes_to_deadline": 103.0}
(MainProcess pid=35900) INFO 10-31 03:58:49 [loggers.py:335] Engine 000: Avg prompt throughput: 335.7 tokens/s, Avg generation throughput: 3902.5 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 91.5%, Prefix cache hit rate: 42.6%
progress {"time": "03:59:49", "elapsed_min": 40.0, "submitted": 963, "finished": 707, "in_flight": 256, "gen_tokens": 8463387, "tok_per_s": 3526, "chain_tokens_median": 1851.0, "chain_tokens_p90": 6489.900000000001, "chain_tokens_max": 32583, "parse_rate": 0.9954031117397454, "looped": 13, "minutes_to_deadline": 102.0}
(MainProcess pid=35900) INFO 10-31 03:59:49 [loggers.py:335] Engine 000: Avg prompt throughput: 340.6 tokens/s, Avg generation throughput: 3774.5 tokens/s, Running: 256 reqs, Waiting: 642 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 42.9%
progress {"time": "04:00:49", "elapsed_min": 41.0, "submitted": 978, "finished": 722, "in_flight": 256, "gen_tokens": 8668835, "tok_per_s": 3524, "chain_tokens_median": 1849.0, "chain_tokens_p90": 6489.900000000001, "chain_tokens_max": 32630, "parse_rate": 0.9948060941828255, "looped": 15, "minutes_to_deadline": 101.0}
(MainProcess pid=35900) INFO 10-31 04:00:49 [loggers.py:335] Engine 000: Avg prompt throughput: 275.4 tokens/s, Avg generation throughput: 3732.1 tokens/s, Running: 256 reqs, Waiting: 646 reqs, GPU KV cache usage: 96.1%, Prefix cache hit rate: 42.8%
progress {"time": "04:01:49", "elapsed_min": 42.0, "submitted": 992, "finished": 736, "in_flight": 256, "gen_tokens": 8781926, "tok_per_s": 3485, "chain_tokens_median": 1820.0, "chain_tokens_p90": 6483.600000000002, "chain_tokens_max": 32630, "parse_rate": 0.9949048913043478, "looped": 15, "minutes_to_deadline": 100.0}
(MainProcess pid=35900) INFO 10-31 04:01:49 [loggers.py:335] Engine 000: Avg prompt throughput: 364.0 tokens/s, Avg generation throughput: 3638.0 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 42.2%
progress {"time": "04:02:50", "elapsed_min": 43.0, "submitted": 1007, "finished": 751, "in_flight": 256, "gen_tokens": 9016631, "tok_per_s": 3495, "chain_tokens_median": 1819.0, "chain_tokens_p90": 6497.200000000002, "chain_tokens_max": 32630, "parse_rate": 0.9943408788282291, "looped": 17, "minutes_to_deadline": 99.0}
(MainProcess pid=35900) INFO 10-31 04:02:50 [loggers.py:335] Engine 000: Avg prompt throughput: 319.1 tokens/s, Avg generation throughput: 3616.6 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 96.8%, Prefix cache hit rate: 42.0%
progress {"time": "04:03:50", "elapsed_min": 44.0, "submitted": 1024, "finished": 768, "in_flight": 256, "gen_tokens": 9255489, "tok_per_s": 3506, "chain_tokens_median": 1806.5, "chain_tokens_p90": 6511.9, "chain_tokens_max": 32655, "parse_rate": 0.994140625, "looped": 18, "minutes_to_deadline": 98.0}
(MainProcess pid=35900) INFO 10-31 04:03:50 [loggers.py:335] Engine 000: Avg prompt throughput: 389.2 tokens/s, Avg generation throughput: 3680.6 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 41.4%
progress {"time": "04:04:50", "elapsed_min": 45.0, "submitted": 1045, "finished": 789, "in_flight": 256, "gen_tokens": 9556145, "tok_per_s": 3539, "chain_tokens_median": 1821.5, "chain_tokens_p90": 6515.0, "chain_tokens_max": 32655, "parse_rate": 0.993979721166033, "looped": 19, "minutes_to_deadline": 97.0}
(MainProcess pid=35900) INFO 10-31 04:04:50 [loggers.py:335] Engine 000: Avg prompt throughput: 407.9 tokens/s, Avg generation throughput: 3659.1 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 93.9%, Prefix cache hit rate: 40.8%
progress {"time": "04:05:50", "elapsed_min": 46.0, "submitted": 1065, "finished": 809, "in_flight": 256, "gen_tokens": 9842770, "tok_per_s": 3566, "chain_tokens_median": 1815.0, "chain_tokens_p90": 6559.5, "chain_tokens_max": 32655, "parse_rate": 0.9935105067985167, "looped": 21, "minutes_to_deadline": 96.0}
(MainProcess pid=35900) INFO 10-31 04:05:50 [loggers.py:335] Engine 000: Avg prompt throughput: 468.4 tokens/s, Avg generation throughput: 3667.6 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 91.0%, Prefix cache hit rate: 40.1%
progress {"time": "04:06:50", "elapsed_min": 47.0, "submitted": 1079, "finished": 823, "in_flight": 256, "gen_tokens": 10018049, "tok_per_s": 3552, "chain_tokens_median": 1806.5, "chain_tokens_p90": 6568.9, "chain_tokens_max": 32655, "parse_rate": 0.9933171324422844, "looped": 22, "minutes_to_deadline": 95.0}
(MainProcess pid=35900) INFO 10-31 04:06:50 [loggers.py:335] Engine 000: Avg prompt throughput: 341.0 tokens/s, Avg generation throughput: 3876.9 tokens/s, Running: 256 reqs, Waiting: 643 reqs, GPU KV cache usage: 94.6%, Prefix cache hit rate: 40.0%
progress {"time": "04:07:50", "elapsed_min": 48.0, "submitted": 1091, "finished": 835, "in_flight": 256, "gen_tokens": 10166900, "tok_per_s": 3530, "chain_tokens_median": 1806.0, "chain_tokens_p90": 6569.1, "chain_tokens_max": 32655, "parse_rate": 0.9934131736526947, "looped": 22, "minutes_to_deadline": 94.0}
(MainProcess pid=35900) INFO 10-31 04:07:50 [loggers.py:335] Engine 000: Avg prompt throughput: 346.6 tokens/s, Avg generation throughput: 3795.8 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 97.7%, Prefix cache hit rate: 40.4%
progress {"time": "04:08:50", "elapsed_min": 49.0, "submitted": 1104, "finished": 848, "in_flight": 256, "gen_tokens": 10348818, "tok_per_s": 3520, "chain_tokens_median": 1805.5, "chain_tokens_p90": 6592.8, "chain_tokens_max": 32655, "parse_rate": 0.9932193396226415, "looped": 23, "minutes_to_deadline": 93.0}
(MainProcess pid=35900) INFO 10-31 04:08:50 [loggers.py:335] Engine 000: Avg prompt throughput: 318.9 tokens/s, Avg generation throughput: 3663.4 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 95.6%, Prefix cache hit rate: 38.0%
progress {"time": "04:09:50", "elapsed_min": 50.0, "submitted": 1117, "finished": 861, "in_flight": 256, "gen_tokens": 10524016, "tok_per_s": 3508, "chain_tokens_median": 1819.0, "chain_tokens_p90": 6583.200000000002, "chain_tokens_max": 32655, "parse_rate": 0.993321718931475, "looped": 23, "minutes_to_deadline": 92.0}
(MainProcess pid=35900) INFO 10-31 04:09:50 [loggers.py:335] Engine 000: Avg prompt throughput: 356.5 tokens/s, Avg generation throughput: 3701.9 tokens/s, Running: 256 reqs, Waiting: 609 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 38.2%
progress {"time": "04:10:50", "elapsed_min": 51.0, "submitted": 1132, "finished": 876, "in_flight": 256, "gen_tokens": 10729934, "tok_per_s": 3506, "chain_tokens_median": 1841.5, "chain_tokens_p90": 6569.700000000001, "chain_tokens_max": 32655, "parse_rate": 0.9931506849315068, "looped": 24, "minutes_to_deadline": 91.0}
(MainProcess pid=35900) INFO 10-31 04:10:50 [loggers.py:335] Engine 000: Avg prompt throughput: 268.1 tokens/s, Avg generation throughput: 3605.5 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 97.9%, Prefix cache hit rate: 38.5%
progress {"time": "04:11:50", "elapsed_min": 52.0, "submitted": 1151, "finished": 895, "in_flight": 256, "gen_tokens": 11031197, "tok_per_s": 3535, "chain_tokens_median": 1862.0, "chain_tokens_p90": 6619.699999999999, "chain_tokens_max": 32655, "parse_rate": 0.9930167597765364, "looped": 25, "minutes_to_deadline": 90.0}
(MainProcess pid=35900) INFO 10-31 04:11:50 [loggers.py:335] Engine 000: Avg prompt throughput: 360.9 tokens/s, Avg generation throughput: 3667.8 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 98.8%, Prefix cache hit rate: 38.2%
progress {"time": "04:12:50", "elapsed_min": 53.0, "submitted": 1166, "finished": 910, "in_flight": 256, "gen_tokens": 11272904, "tok_per_s": 3545, "chain_tokens_median": 1879.5, "chain_tokens_p90": 6643.699999999999, "chain_tokens_max": 32655, "parse_rate": 0.9925824175824176, "looped": 27, "minutes_to_deadline": 89.0}
(MainProcess pid=35900) INFO 10-31 04:12:50 [loggers.py:335] Engine 000: Avg prompt throughput: 362.6 tokens/s, Avg generation throughput: 3637.8 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 38.8%
progress {"time": "04:13:50", "elapsed_min": 54.0, "submitted": 1184, "finished": 928, "in_flight": 256, "gen_tokens": 11499625, "tok_per_s": 3549, "chain_tokens_median": 1898.5, "chain_tokens_p90": 6618.0, "chain_tokens_max": 32655, "parse_rate": 0.9924568965517241, "looped": 28, "minutes_to_deadline": 88.0}
(MainProcess pid=35900) INFO 10-31 04:13:50 [loggers.py:335] Engine 000: Avg prompt throughput: 251.1 tokens/s, Avg generation throughput: 3765.9 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 97.6%, Prefix cache hit rate: 39.0%
progress {"time": "04:14:50", "elapsed_min": 55.0, "submitted": 1200, "finished": 944, "in_flight": 256, "gen_tokens": 11710817, "tok_per_s": 3548, "chain_tokens_median": 1910.0, "chain_tokens_p90": 6618.0, "chain_tokens_max": 32655, "parse_rate": 0.9923199152542372, "looped": 29, "minutes_to_deadline": 87.0}
(MainProcess pid=35900) INFO 10-31 04:14:50 [loggers.py:335] Engine 000: Avg prompt throughput: 354.5 tokens/s, Avg generation throughput: 3650.7 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 39.3%
progress {"time": "04:15:50", "elapsed_min": 56.0, "submitted": 1211, "finished": 955, "in_flight": 256, "gen_tokens": 11871148, "tok_per_s": 3533, "chain_tokens_median": 1909.5, "chain_tokens_p90": 6618.0, "chain_tokens_max": 32655, "parse_rate": 0.9921465968586387, "looped": 30, "minutes_to_deadline": 86.0}
(MainProcess pid=35900) INFO 10-31 04:15:50 [loggers.py:335] Engine 000: Avg prompt throughput: 359.6 tokens/s, Avg generation throughput: 3659.3 tokens/s, Running: 256 reqs, Waiting: 619 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 38.0%
progress {"time": "04:16:50", "elapsed_min": 57.0, "submitted": 1220, "finished": 964, "in_flight": 256, "gen_tokens": 11960124, "tok_per_s": 3497, "chain_tokens_median": 1906.0, "chain_tokens_p90": 6592.0, "chain_tokens_max": 32655, "parse_rate": 0.9922199170124482, "looped": 30, "minutes_to_deadline": 85.0}
(MainProcess pid=35900) INFO 10-31 04:16:50 [loggers.py:335] Engine 000: Avg prompt throughput: 132.4 tokens/s, Avg generation throughput: 2660.8 tokens/s, Running: 237 reqs, Waiting: 640 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 38.6%
progress {"time": "04:17:50", "elapsed_min": 58.0, "submitted": 1231, "finished": 975, "in_flight": 256, "gen_tokens": 12138938, "tok_per_s": 3488, "chain_tokens_median": 1912.5, "chain_tokens_p90": 6585.599999999999, "chain_tokens_max": 32661, "parse_rate": 0.9917948717948718, "looped": 32, "minutes_to_deadline": 84.0}
(MainProcess pid=35900) INFO 10-31 04:17:50 [loggers.py:335] Engine 000: Avg prompt throughput: 195.4 tokens/s, Avg generation throughput: 2792.2 tokens/s, Running: 239 reqs, Waiting: 640 reqs, GPU KV cache usage: 99.4%, Prefix cache hit rate: 38.6%
progress {"time": "04:18:50", "elapsed_min": 59.0, "submitted": 1244, "finished": 988, "in_flight": 256, "gen_tokens": 12267032, "tok_per_s": 3465, "chain_tokens_median": 1914.0, "chain_tokens_p90": 6566.4000000000015, "chain_tokens_max": 32661, "parse_rate": 0.9919028340080972, "looped": 32, "minutes_to_deadline": 83.0}
(MainProcess pid=35900) INFO 10-31 04:18:50 [loggers.py:335] Engine 000: Avg prompt throughput: 268.7 tokens/s, Avg generation throughput: 2550.4 tokens/s, Running: 240 reqs, Waiting: 640 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 38.9%
progress {"time": "04:19:50", "elapsed_min": 60.0, "submitted": 1258, "finished": 1002, "in_flight": 256, "gen_tokens": 12529411, "tok_per_s": 3480, "chain_tokens_median": 1927.0, "chain_tokens_p90": 6556.800000000003, "chain_tokens_max": 32661, "parse_rate": 0.9910179640718563, "looped": 36, "minutes_to_deadline": 82.0}
(MainProcess pid=35900) INFO 10-31 04:19:50 [loggers.py:335] Engine 000: Avg prompt throughput: 356.1 tokens/s, Avg generation throughput: 3663.6 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 97.1%, Prefix cache hit rate: 38.7%
progress {"time": "04:20:50", "elapsed_min": 61.0, "submitted": 1276, "finished": 1020, "in_flight": 256, "gen_tokens": 12767733, "tok_per_s": 3488, "chain_tokens_median": 1935.0, "chain_tokens_p90": 6568.1, "chain_tokens_max": 32667, "parse_rate": 0.9909313725490196, "looped": 37, "minutes_to_deadline": 81.0}
(MainProcess pid=35900) INFO 10-31 04:20:50 [loggers.py:335] Engine 000: Avg prompt throughput: 355.2 tokens/s, Avg generation throughput: 3641.1 tokens/s, Running: 255 reqs, Waiting: 615 reqs, GPU KV cache usage: 94.4%, Prefix cache hit rate: 38.0%
progress {"time": "04:21:50", "elapsed_min": 62.0, "submitted": 1298, "finished": 1042, "in_flight": 256, "gen_tokens": 13042289, "tok_per_s": 3506, "chain_tokens_median": 1938.5, "chain_tokens_p90": 6558.900000000002, "chain_tokens_max": 32667, "parse_rate": 0.9911228406909789, "looped": 37, "minutes_to_deadline": 80.0}
(MainProcess pid=35900) INFO 10-31 04:21:50 [loggers.py:335] Engine 000: Avg prompt throughput: 495.7 tokens/s, Avg generation throughput: 3645.9 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 95.0%, Prefix cache hit rate: 40.6%
progress {"time": "04:22:50", "elapsed_min": 63.0, "submitted": 1317, "finished": 1061, "in_flight": 256, "gen_tokens": 13432876, "tok_per_s": 3553, "chain_tokens_median": 1954.0, "chain_tokens_p90": 6628.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9903393025447691, "looped": 41, "minutes_to_deadline": 79.0}
(MainProcess pid=35900) INFO 10-31 04:22:50 [loggers.py:335] Engine 000: Avg prompt throughput: 418.0 tokens/s, Avg generation throughput: 3765.6 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 90.4%, Prefix cache hit rate: 42.1%
progress {"time": "04:23:50", "elapsed_min": 64.0, "submitted": 1333, "finished": 1077, "in_flight": 256, "gen_tokens": 13646967, "tok_per_s": 3554, "chain_tokens_median": 1942.0, "chain_tokens_p90": 6651.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9902506963788301, "looped": 42, "minutes_to_deadline": 78.0}
(MainProcess pid=35900) INFO 10-31 04:23:50 [loggers.py:335] Engine 000: Avg prompt throughput: 278.1 tokens/s, Avg generation throughput: 3847.1 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 92.2%, Prefix cache hit rate: 41.5%
progress {"time": "04:24:50", "elapsed_min": 65.0, "submitted": 1345, "finished": 1089, "in_flight": 256, "gen_tokens": 13820340, "tok_per_s": 3543, "chain_tokens_median": 1948.0, "chain_tokens_p90": 6652.0, "chain_tokens_max": 32667, "parse_rate": 0.9901285583103765, "looped": 43, "minutes_to_deadline": 77.0}
(MainProcess pid=35900) INFO 10-31 04:24:50 [loggers.py:335] Engine 000: Avg prompt throughput: 445.3 tokens/s, Avg generation throughput: 3722.9 tokens/s, Running: 255 reqs, Waiting: 621 reqs, GPU KV cache usage: 93.9%, Prefix cache hit rate: 43.2%
progress {"time": "04:25:50", "elapsed_min": 66.0, "submitted": 1362, "finished": 1106, "in_flight": 256, "gen_tokens": 14013295, "tok_per_s": 3538, "chain_tokens_median": 1933.0, "chain_tokens_p90": 6636.300000000003, "chain_tokens_max": 32667, "parse_rate": 0.9898282097649186, "looped": 45, "minutes_to_deadline": 76.0}
(MainProcess pid=35900) INFO 10-31 04:25:50 [loggers.py:335] Engine 000: Avg prompt throughput: 414.3 tokens/s, Avg generation throughput: 3718.6 tokens/s, Running: 255 reqs, Waiting: 619 reqs, GPU KV cache usage: 93.2%, Prefix cache hit rate: 44.2%
progress {"time": "04:26:50", "elapsed_min": 67.0, "submitted": 1386, "finished": 1130, "in_flight": 256, "gen_tokens": 14291902, "tok_per_s": 3555, "chain_tokens_median": 1937.0, "chain_tokens_p90": 6627.299999999999, "chain_tokens_max": 32667, "parse_rate": 0.9900442477876106, "looped": 45, "minutes_to_deadline": 75.0}
(MainProcess pid=35900) INFO 10-31 04:26:50 [loggers.py:335] Engine 000: Avg prompt throughput: 397.6 tokens/s, Avg generation throughput: 3763.7 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 92.3%, Prefix cache hit rate: 44.8%
progress {"time": "04:27:50", "elapsed_min": 68.0, "submitted": 1401, "finished": 1145, "in_flight": 256, "gen_tokens": 14558872, "tok_per_s": 3568, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6654.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.990174672489083, "looped": 45, "minutes_to_deadline": 74.0}
(MainProcess pid=35900) INFO 10-31 04:27:50 [loggers.py:335] Engine 000: Avg prompt throughput: 279.7 tokens/s, Avg generation throughput: 3821.4 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 92.6%, Prefix cache hit rate: 44.9%
progress {"time": "04:28:50", "elapsed_min": 69.0, "submitted": 1412, "finished": 1156, "in_flight": 256, "gen_tokens": 14703234, "tok_per_s": 3551, "chain_tokens_median": 1948.0, "chain_tokens_p90": 6656.7, "chain_tokens_max": 32667, "parse_rate": 0.9900519031141869, "looped": 46, "minutes_to_deadline": 73.0}
(MainProcess pid=35900) INFO 10-31 04:28:50 [loggers.py:335] Engine 000: Avg prompt throughput: 362.5 tokens/s, Avg generation throughput: 3833.6 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 95.5%, Prefix cache hit rate: 45.4%
progress {"time": "04:29:50", "elapsed_min": 70.0, "submitted": 1429, "finished": 1173, "in_flight": 256, "gen_tokens": 14938583, "tok_per_s": 3556, "chain_tokens_median": 1977.5, "chain_tokens_p90": 6649.300000000004, "chain_tokens_max": 32667, "parse_rate": 0.9899829497016198, "looped": 47, "minutes_to_deadline": 72.0}
(MainProcess pid=35900) INFO 10-31 04:29:50 [loggers.py:335] Engine 000: Avg prompt throughput: 368.4 tokens/s, Avg generation throughput: 3748.4 tokens/s, Running: 255 reqs, Waiting: 645 reqs, GPU KV cache usage: 96.5%, Prefix cache hit rate: 45.8%
progress {"time": "04:30:50", "elapsed_min": 71.0, "submitted": 1441, "finished": 1185, "in_flight": 256, "gen_tokens": 15037877, "tok_per_s": 3530, "chain_tokens_median": 1960.0, "chain_tokens_p90": 6626.1, "chain_tokens_max": 32667, "parse_rate": 0.990084388185654, "looped": 47, "minutes_to_deadline": 71.0}
(MainProcess pid=35900) INFO 10-31 04:30:50 [loggers.py:335] Engine 000: Avg prompt throughput: 371.3 tokens/s, Avg generation throughput: 3628.8 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 44.8%
progress {"time": "04:31:50", "elapsed_min": 72.0, "submitted": 1456, "finished": 1200, "in_flight": 256, "gen_tokens": 15179402, "tok_per_s": 3513, "chain_tokens_median": 1949.5, "chain_tokens_p90": 6618.1, "chain_tokens_max": 32667, "parse_rate": 0.9902083333333334, "looped": 47, "minutes_to_deadline": 70.0}
(MainProcess pid=35900) INFO 10-31 04:31:50 [loggers.py:335] Engine 000: Avg prompt throughput: 284.2 tokens/s, Avg generation throughput: 3543.6 tokens/s, Running: 250 reqs, Waiting: 628 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 45.7%
progress {"time": "04:32:50", "elapsed_min": 73.0, "submitted": 1468, "finished": 1212, "in_flight": 256, "gen_tokens": 15480680, "tok_per_s": 3534, "chain_tokens_median": 1974.5, "chain_tokens_p90": 6651.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.989480198019802, "looped": 51, "minutes_to_deadline": 69.0}
(MainProcess pid=35900) INFO 10-31 04:32:50 [loggers.py:335] Engine 000: Avg prompt throughput: 197.4 tokens/s, Avg generation throughput: 3039.5 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 94.9%, Prefix cache hit rate: 45.4%
progress {"time": "04:33:50", "elapsed_min": 74.0, "submitted": 1487, "finished": 1231, "in_flight": 256, "gen_tokens": 15743030, "tok_per_s": 3545, "chain_tokens_median": 1985.0, "chain_tokens_p90": 6623.899999999999, "chain_tokens_max": 32667, "parse_rate": 0.9892363931762794, "looped": 53, "minutes_to_deadline": 68.0}
(MainProcess pid=35900) INFO 10-31 04:33:50 [loggers.py:335] Engine 000: Avg prompt throughput: 515.6 tokens/s, Avg generation throughput: 3616.0 tokens/s, Running: 256 reqs, Waiting: 615 reqs, GPU KV cache usage: 90.0%, Prefix cache hit rate: 45.3%
progress {"time": "04:34:50", "elapsed_min": 75.0, "submitted": 1504, "finished": 1248, "in_flight": 256, "gen_tokens": 16016777, "tok_per_s": 3559, "chain_tokens_median": 2000.0, "chain_tokens_p90": 6638.100000000005, "chain_tokens_max": 32667, "parse_rate": 0.9891826923076923, "looped": 54, "minutes_to_deadline": 67.0}
(MainProcess pid=35900) INFO 10-31 04:34:50 [loggers.py:335] Engine 000: Avg prompt throughput: 279.1 tokens/s, Avg generation throughput: 3808.2 tokens/s, Running: 256 reqs, Waiting: 608 reqs, GPU KV cache usage: 90.0%, Prefix cache hit rate: 45.7%
progress {"time": "04:35:50", "elapsed_min": 76.0, "submitted": 1527, "finished": 1271, "in_flight": 256, "gen_tokens": 16324688, "tok_per_s": 3580, "chain_tokens_median": 2000.0, "chain_tokens_p90": 6655.4, "chain_tokens_max": 32667, "parse_rate": 0.9891817466561762, "looped": 55, "minutes_to_deadline": 66.0}
(MainProcess pid=35900) INFO 10-31 04:35:50 [loggers.py:335] Engine 000: Avg prompt throughput: 346.3 tokens/s, Avg generation throughput: 3868.3 tokens/s, Running: 256 reqs, Waiting: 642 reqs, GPU KV cache usage: 90.3%, Prefix cache hit rate: 45.4%
progress {"time": "04:36:50", "elapsed_min": 77.0, "submitted": 1544, "finished": 1288, "in_flight": 256, "gen_tokens": 16533380, "tok_per_s": 3578, "chain_tokens_median": 2001.0, "chain_tokens_p90": 6653.600000000002, "chain_tokens_max": 32667, "parse_rate": 0.9891304347826086, "looped": 56, "minutes_to_deadline": 65.0}
(MainProcess pid=35900) INFO 10-31 04:36:50 [loggers.py:335] Engine 000: Avg prompt throughput: 362.0 tokens/s, Avg generation throughput: 3816.0 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 89.9%, Prefix cache hit rate: 44.9%
progress {"time": "04:37:50", "elapsed_min": 78.0, "submitted": 1564, "finished": 1308, "in_flight": 256, "gen_tokens": 16856942, "tok_per_s": 3602, "chain_tokens_median": 2001.0, "chain_tokens_p90": 6658.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.988914373088685, "looped": 58, "minutes_to_deadline": 64.0}
(MainProcess pid=35900) INFO 10-31 04:37:50 [loggers.py:335] Engine 000: Avg prompt throughput: 427.3 tokens/s, Avg generation throughput: 3778.5 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 43.4%
progress {"time": "04:38:50", "elapsed_min": 79.0, "submitted": 1580, "finished": 1324, "in_flight": 256, "gen_tokens": 17004682, "tok_per_s": 3587, "chain_tokens_median": 2001.0, "chain_tokens_p90": 6641.0, "chain_tokens_max": 32667, "parse_rate": 0.9890483383685801, "looped": 58, "minutes_to_deadline": 63.0}
(MainProcess pid=35900) INFO 10-31 04:38:50 [loggers.py:335] Engine 000: Avg prompt throughput: 350.7 tokens/s, Avg generation throughput: 3880.9 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 89.6%, Prefix cache hit rate: 43.2%
progress {"time": "04:39:50", "elapsed_min": 80.0, "submitted": 1596, "finished": 1340, "in_flight": 256, "gen_tokens": 17225887, "tok_per_s": 3588, "chain_tokens_median": 1997.0, "chain_tokens_p90": 6643.700000000003, "chain_tokens_max": 32667, "parse_rate": 0.9888059701492538, "looped": 60, "minutes_to_deadline": 62.0}
(MainProcess pid=35900) INFO 10-31 04:39:50 [loggers.py:335] Engine 000: Avg prompt throughput: 388.3 tokens/s, Avg generation throughput: 3832.3 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 88.7%, Prefix cache hit rate: 41.5%
progress {"time": "04:40:50", "elapsed_min": 81.0, "submitted": 1616, "finished": 1360, "in_flight": 256, "gen_tokens": 17502977, "tok_per_s": 3601, "chain_tokens_median": 1995.5, "chain_tokens_p90": 6650.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9887867647058823, "looped": 61, "minutes_to_deadline": 61.0}
(MainProcess pid=35900) INFO 10-31 04:40:50 [loggers.py:335] Engine 000: Avg prompt throughput: 402.5 tokens/s, Avg generation throughput: 3824.8 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 86.1%, Prefix cache hit rate: 39.1%
progress {"time": "04:41:50", "elapsed_min": 82.0, "submitted": 1640, "finished": 1384, "in_flight": 256, "gen_tokens": 17780909, "tok_per_s": 3614, "chain_tokens_median": 1997.0, "chain_tokens_p90": 6628.5, "chain_tokens_max": 32667, "parse_rate": 0.9888005780346821, "looped": 62, "minutes_to_deadline": 60.0}
(MainProcess pid=35900) INFO 10-31 04:41:50 [loggers.py:335] Engine 000: Avg prompt throughput: 425.1 tokens/s, Avg generation throughput: 3876.4 tokens/s, Running: 255 reqs, Waiting: 638 reqs, GPU KV cache usage: 86.8%, Prefix cache hit rate: 37.7%
progress {"time": "04:42:50", "elapsed_min": 83.0, "submitted": 1652, "finished": 1396, "in_flight": 256, "gen_tokens": 17890485, "tok_per_s": 3592, "chain_tokens_median": 1986.0, "chain_tokens_p90": 6626.7, "chain_tokens_max": 32667, "parse_rate": 0.9888968481375359, "looped": 62, "minutes_to_deadline": 59.0}
(MainProcess pid=35900) INFO 10-31 04:42:50 [loggers.py:335] Engine 000: Avg prompt throughput: 319.0 tokens/s, Avg generation throughput: 3851.3 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 89.4%, Prefix cache hit rate: 37.5%
progress {"time": "04:43:50", "elapsed_min": 84.0, "submitted": 1673, "finished": 1417, "in_flight": 256, "gen_tokens": 18237221, "tok_per_s": 3618, "chain_tokens_median": 1985.0, "chain_tokens_p90": 6640.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9885321100917431, "looped": 65, "minutes_to_deadline": 58.0}
(MainProcess pid=35900) INFO 10-31 04:43:50 [loggers.py:335] Engine 000: Avg prompt throughput: 349.9 tokens/s, Avg generation throughput: 3872.2 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 83.5%, Prefix cache hit rate: 35.4%
progress {"time": "04:44:50", "elapsed_min": 85.0, "submitted": 1697, "finished": 1441, "in_flight": 256, "gen_tokens": 18480870, "tok_per_s": 3623, "chain_tokens_median": 1977.5, "chain_tokens_p90": 6599.299999999998, "chain_tokens_max": 32667, "parse_rate": 0.9887231089521166, "looped": 65, "minutes_to_deadline": 57.0}
(MainProcess pid=35900) INFO 10-31 04:44:50 [loggers.py:335] Engine 000: Avg prompt throughput: 569.4 tokens/s, Avg generation throughput: 3876.4 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 82.8%, Prefix cache hit rate: 35.8%
progress {"time": "04:45:50", "elapsed_min": 86.0, "submitted": 1717, "finished": 1461, "in_flight": 256, "gen_tokens": 18658120, "tok_per_s": 3615, "chain_tokens_median": 1961.5, "chain_tokens_p90": 6583.199999999999, "chain_tokens_max": 32667, "parse_rate": 0.9888774811772758, "looped": 65, "minutes_to_deadline": 56.0}
(MainProcess pid=35900) INFO 10-31 04:45:50 [loggers.py:335] Engine 000: Avg prompt throughput: 580.3 tokens/s, Avg generation throughput: 3842.5 tokens/s, Running: 256 reqs, Waiting: 646 reqs, GPU KV cache usage: 86.8%, Prefix cache hit rate: 39.2%
progress {"time": "04:46:50", "elapsed_min": 87.0, "submitted": 1739, "finished": 1483, "in_flight": 256, "gen_tokens": 19007559, "tok_per_s": 3641, "chain_tokens_median": 1973.5, "chain_tokens_p90": 6592.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9887053270397842, "looped": 67, "minutes_to_deadline": 55.0}
(MainProcess pid=35900) INFO 10-31 04:46:50 [loggers.py:335] Engine 000: Avg prompt throughput: 324.8 tokens/s, Avg generation throughput: 3863.6 tokens/s, Running: 256 reqs, Waiting: 656 reqs, GPU KV cache usage: 85.0%, Prefix cache hit rate: 39.5%
progress {"time": "04:47:50", "elapsed_min": 88.0, "submitted": 1758, "finished": 1502, "in_flight": 256, "gen_tokens": 19187177, "tok_per_s": 3634, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6569.3, "chain_tokens_max": 32667, "parse_rate": 0.9888482023968043, "looped": 67, "minutes_to_deadline": 54.0}
(MainProcess pid=35900) INFO 10-31 04:47:50 [loggers.py:335] Engine 000: Avg prompt throughput: 342.4 tokens/s, Avg generation throughput: 3906.4 tokens/s, Running: 256 reqs, Waiting: 652 reqs, GPU KV cache usage: 84.4%, Prefix cache hit rate: 39.2%
progress {"time": "04:48:50", "elapsed_min": 89.0, "submitted": 1777, "finished": 1521, "in_flight": 256, "gen_tokens": 19478374, "tok_per_s": 3647, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6599.299999999998, "chain_tokens_max": 32667, "parse_rate": 0.9886587771203156, "looped": 69, "minutes_to_deadline": 53.0}
(MainProcess pid=35900) INFO 10-31 04:48:50 [loggers.py:335] Engine 000: Avg prompt throughput: 432.5 tokens/s, Avg generation throughput: 3944.2 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 82.9%, Prefix cache hit rate: 39.8%
progress {"time": "04:49:50", "elapsed_min": 90.0, "submitted": 1793, "finished": 1537, "in_flight": 256, "gen_tokens": 19708115, "tok_per_s": 3649, "chain_tokens_median": 1966.0, "chain_tokens_p90": 6603.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9886141834743006, "looped": 70, "minutes_to_deadline": 52.0}
(MainProcess pid=35900) INFO 10-31 04:49:50 [loggers.py:335] Engine 000: Avg prompt throughput: 544.4 tokens/s, Avg generation throughput: 3923.4 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 82.3%, Prefix cache hit rate: 40.7%
progress {"time": "04:50:50", "elapsed_min": 91.0, "submitted": 1813, "finished": 1557, "in_flight": 256, "gen_tokens": 19881265, "tok_per_s": 3641, "chain_tokens_median": 1947.0, "chain_tokens_p90": 6585.0, "chain_tokens_max": 32667, "parse_rate": 0.9887604367373154, "looped": 70, "minutes_to_deadline": 51.0}
(MainProcess pid=35900) INFO 10-31 04:50:50 [loggers.py:335] Engine 000: Avg prompt throughput: 480.5 tokens/s, Avg generation throughput: 3897.5 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 83.4%, Prefix cache hit rate: 41.0%
progress {"time": "04:51:50", "elapsed_min": 92.0, "submitted": 1832, "finished": 1576, "in_flight": 256, "gen_tokens": 20101026, "tok_per_s": 3641, "chain_tokens_median": 1946.5, "chain_tokens_p90": 6578.4, "chain_tokens_max": 32667, "parse_rate": 0.9888959390862944, "looped": 70, "minutes_to_deadline": 50.0}
(MainProcess pid=35900) INFO 10-31 04:51:50 [loggers.py:335] Engine 000: Avg prompt throughput: 281.9 tokens/s, Avg generation throughput: 4081.4 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 84.4%, Prefix cache hit rate: 41.6%
progress {"time": "04:52:50", "elapsed_min": 93.0, "submitted": 1850, "finished": 1594, "in_flight": 256, "gen_tokens": 20312937, "tok_per_s": 3640, "chain_tokens_median": 1950.5, "chain_tokens_p90": 6584.0, "chain_tokens_max": 32667, "parse_rate": 0.989021329987453, "looped": 70, "minutes_to_deadline": 49.0}
(MainProcess pid=35900) INFO 10-31 04:52:50 [loggers.py:335] Engine 000: Avg prompt throughput: 270.4 tokens/s, Avg generation throughput: 3953.5 tokens/s, Running: 256 reqs, Waiting: 647 reqs, GPU KV cache usage: 87.4%, Prefix cache hit rate: 42.3%
progress {"time": "04:53:50", "elapsed_min": 94.0, "submitted": 1870, "finished": 1614, "in_flight": 256, "gen_tokens": 20642847, "tok_per_s": 3660, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6597.5, "chain_tokens_max": 32667, "parse_rate": 0.989002478314746, "looped": 71, "minutes_to_deadline": 48.0}
(MainProcess pid=35900) INFO 10-31 04:53:50 [loggers.py:335] Engine 000: Avg prompt throughput: 438.0 tokens/s, Avg generation throughput: 3765.4 tokens/s, Running: 256 reqs, Waiting: 649 reqs, GPU KV cache usage: 85.8%, Prefix cache hit rate: 43.5%
progress {"time": "04:54:50", "elapsed_min": 95.0, "submitted": 1888, "finished": 1632, "in_flight": 256, "gen_tokens": 20868168, "tok_per_s": 3661, "chain_tokens_median": 1952.5, "chain_tokens_p90": 6618.0, "chain_tokens_max": 32667, "parse_rate": 0.9891237745098039, "looped": 71, "minutes_to_deadline": 47.0}
(MainProcess pid=35900) INFO 10-31 04:54:50 [loggers.py:335] Engine 000: Avg prompt throughput: 429.3 tokens/s, Avg generation throughput: 3867.5 tokens/s, Running: 256 reqs, Waiting: 654 reqs, GPU KV cache usage: 87.8%, Prefix cache hit rate: 44.0%
progress {"time": "04:55:50", "elapsed_min": 96.0, "submitted": 1904, "finished": 1648, "in_flight": 256, "gen_tokens": 21111739, "tok_per_s": 3665, "chain_tokens_median": 1951.5, "chain_tokens_p90": 6601.100000000005, "chain_tokens_max": 32667, "parse_rate": 0.9887742718446602, "looped": 74, "minutes_to_deadline": 46.0}
(MainProcess pid=35900) INFO 10-31 04:55:50 [loggers.py:335] Engine 000: Avg prompt throughput: 464.3 tokens/s, Avg generation throughput: 3799.8 tokens/s, Running: 256 reqs, Waiting: 646 reqs, GPU KV cache usage: 85.4%, Prefix cache hit rate: 44.5%
progress {"time": "04:56:50", "elapsed_min": 97.0, "submitted": 1922, "finished": 1666, "in_flight": 256, "gen_tokens": 21387563, "tok_per_s": 3674, "chain_tokens_median": 1967.0, "chain_tokens_p90": 6592.4, "chain_tokens_max": 32667, "parse_rate": 0.9885954381752701, "looped": 76, "minutes_to_deadline": 45.0}
(MainProcess pid=35900) INFO 10-31 04:56:50 [loggers.py:335] Engine 000: Avg prompt throughput: 278.0 tokens/s, Avg generation throughput: 4004.7 tokens/s, Running: 256 reqs, Waiting: 659 reqs, GPU KV cache usage: 84.1%, Prefix cache hit rate: 44.4%
progress {"time": "04:57:50", "elapsed_min": 98.0, "submitted": 1936, "finished": 1680, "in_flight": 256, "gen_tokens": 21572174, "tok_per_s": 3668, "chain_tokens_median": 1983.0, "chain_tokens_p90": 6585.600000000002, "chain_tokens_max": 32667, "parse_rate": 0.9886904761904762, "looped": 76, "minutes_to_deadline": 44.0}
(MainProcess pid=35900) INFO 10-31 04:57:50 [loggers.py:335] Engine 000: Avg prompt throughput: 291.6 tokens/s, Avg generation throughput: 4017.5 tokens/s, Running: 256 reqs, Waiting: 655 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 45.1%
progress {"time": "04:58:50", "elapsed_min": 99.0, "submitted": 1946, "finished": 1690, "in_flight": 256, "gen_tokens": 21687067, "tok_per_s": 3651, "chain_tokens_median": 1977.5, "chain_tokens_p90": 6602.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9887573964497042, "looped": 76, "minutes_to_deadline": 43.0}
(MainProcess pid=35900) INFO 10-31 04:58:50 [loggers.py:335] Engine 000: Avg prompt throughput: 361.5 tokens/s, Avg generation throughput: 3940.7 tokens/s, Running: 256 reqs, Waiting: 633 reqs, GPU KV cache usage: 88.5%, Prefix cache hit rate: 44.3%
progress {"time": "04:59:50", "elapsed_min": 100.0, "submitted": 1964, "finished": 1708, "in_flight": 256, "gen_tokens": 22015887, "tok_per_s": 3669, "chain_tokens_median": 1990.5, "chain_tokens_p90": 6626.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9885831381733021, "looped": 78, "minutes_to_deadline": 42.0}
(MainProcess pid=35900) INFO 10-31 04:59:50 [loggers.py:335] Engine 000: Avg prompt throughput: 371.3 tokens/s, Avg generation throughput: 3944.5 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 86.1%, Prefix cache hit rate: 41.3%
progress {"time": "05:00:50", "elapsed_min": 101.0, "submitted": 1978, "finished": 1722, "in_flight": 256, "gen_tokens": 22123145, "tok_per_s": 3650, "chain_tokens_median": 1970.5, "chain_tokens_p90": 6621.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9886759581881533, "looped": 78, "minutes_to_deadline": 41.0}
(MainProcess pid=35900) INFO 10-31 05:00:50 [loggers.py:335] Engine 000: Avg prompt throughput: 522.1 tokens/s, Avg generation throughput: 3794.8 tokens/s, Running: 256 reqs, Waiting: 612 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 41.7%
progress {"time": "05:01:50", "elapsed_min": 102.0, "submitted": 2001, "finished": 1745, "in_flight": 256, "gen_tokens": 22451966, "tok_per_s": 3668, "chain_tokens_median": 1978.5, "chain_tokens_p90": 6627.0, "chain_tokens_max": 32667, "parse_rate": 0.9888252148997134, "looped": 78, "minutes_to_deadline": 40.0}
(MainProcess pid=35900) INFO 10-31 05:01:50 [loggers.py:335] Engine 000: Avg prompt throughput: 386.5 tokens/s, Avg generation throughput: 3829.2 tokens/s, Running: 255 reqs, Waiting: 628 reqs, GPU KV cache usage: 85.1%, Prefix cache hit rate: 41.9%
progress {"time": "05:02:50", "elapsed_min": 103.0, "submitted": 2022, "finished": 1766, "in_flight": 256, "gen_tokens": 22830935, "tok_per_s": 3694, "chain_tokens_median": 2000.0, "chain_tokens_p90": 6660.4, "chain_tokens_max": 32667, "parse_rate": 0.9888165345413363, "looped": 79, "minutes_to_deadline": 39.0}
(MainProcess pid=35900) INFO 10-31 05:02:50 [loggers.py:335] Engine 000: Avg prompt throughput: 410.6 tokens/s, Avg generation throughput: 3902.0 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 41.6%
progress {"time": "05:03:50", "elapsed_min": 104.0, "submitted": 2044, "finished": 1788, "in_flight": 256, "gen_tokens": 23073636, "tok_per_s": 3697, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6656.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889541387024608, "looped": 79, "minutes_to_deadline": 38.0}
(MainProcess pid=35900) INFO 10-31 05:03:50 [loggers.py:335] Engine 000: Avg prompt throughput: 479.3 tokens/s, Avg generation throughput: 4033.7 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 82.7%, Prefix cache hit rate: 42.4%
progress {"time": "05:04:50", "elapsed_min": 105.0, "submitted": 2060, "finished": 1804, "in_flight": 256, "gen_tokens": 23228172, "tok_per_s": 3687, "chain_tokens_median": 1986.0, "chain_tokens_p90": 6652.0, "chain_tokens_max": 32667, "parse_rate": 0.9890521064301552, "looped": 79, "minutes_to_deadline": 37.0}
(MainProcess pid=35900) INFO 10-31 05:04:50 [loggers.py:335] Engine 000: Avg prompt throughput: 331.4 tokens/s, Avg generation throughput: 4077.2 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 85.5%, Prefix cache hit rate: 41.6%
progress {"time": "05:05:50", "elapsed_min": 106.0, "submitted": 2077, "finished": 1821, "in_flight": 256, "gen_tokens": 23447235, "tok_per_s": 3686, "chain_tokens_median": 1995.5, "chain_tokens_p90": 6655.4, "chain_tokens_max": 32667, "parse_rate": 0.9891543108182317, "looped": 79, "minutes_to_deadline": 36.0}
(MainProcess pid=35900) INFO 10-31 05:05:50 [loggers.py:335] Engine 000: Avg prompt throughput: 282.0 tokens/s, Avg generation throughput: 4038.8 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 41.3%
progress {"time": "05:06:50", "elapsed_min": 107.0, "submitted": 2096, "finished": 1840, "in_flight": 256, "gen_tokens": 23612614, "tok_per_s": 3678, "chain_tokens_median": 1986.0, "chain_tokens_p90": 6627.300000000001, "chain_tokens_max": 32667, "parse_rate": 0.9892663043478261, "looped": 79, "minutes_to_deadline": 35.0}
(MainProcess pid=35900) INFO 10-31 05:06:50 [loggers.py:335] Engine 000: Avg prompt throughput: 410.3 tokens/s, Avg generation throughput: 3902.3 tokens/s, Running: 256 reqs, Waiting: 647 reqs, GPU KV cache usage: 89.4%, Prefix cache hit rate: 40.4%
progress {"time": "05:07:50", "elapsed_min": 108.0, "submitted": 2110, "finished": 1854, "in_flight": 256, "gen_tokens": 23883206, "tok_per_s": 3685, "chain_tokens_median": 2012.5, "chain_tokens_p90": 6648.5, "chain_tokens_max": 32667, "parse_rate": 0.9893473570658037, "looped": 79, "minutes_to_deadline": 34.0}
(MainProcess pid=35900) INFO 10-31 05:07:50 [loggers.py:335] Engine 000: Avg prompt throughput: 224.7 tokens/s, Avg generation throughput: 3949.6 tokens/s, Running: 256 reqs, Waiting: 652 reqs, GPU KV cache usage: 91.7%, Prefix cache hit rate: 40.6%
progress {"time": "05:08:50", "elapsed_min": 109.0, "submitted": 2127, "finished": 1871, "in_flight": 256, "gen_tokens": 24138428, "tok_per_s": 3690, "chain_tokens_median": 2022.5, "chain_tokens_p90": 6636.299999999998, "chain_tokens_max": 32667, "parse_rate": 0.9891769107429182, "looped": 81, "minutes_to_deadline": 33.0}
(MainProcess pid=35900) INFO 10-31 05:08:50 [loggers.py:335] Engine 000: Avg prompt throughput: 377.3 tokens/s, Avg generation throughput: 3829.7 tokens/s, Running: 256 reqs, Waiting: 651 reqs, GPU KV cache usage: 89.8%, Prefix cache hit rate: 40.5%
progress {"time": "05:09:50", "elapsed_min": 110.0, "submitted": 2140, "finished": 1884, "in_flight": 256, "gen_tokens": 24231831, "tok_per_s": 3671, "chain_tokens_median": 2009.0, "chain_tokens_p90": 6623.0, "chain_tokens_max": 32667, "parse_rate": 0.9892515923566879, "looped": 81, "minutes_to_deadline": 32.0}
(MainProcess pid=35900) INFO 10-31 05:09:50 [loggers.py:335] Engine 000: Avg prompt throughput: 260.0 tokens/s, Avg generation throughput: 3799.5 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 94.0%, Prefix cache hit rate: 40.2%
progress {"time": "05:10:50", "elapsed_min": 111.0, "submitted": 2162, "finished": 1906, "in_flight": 256, "gen_tokens": 24572038, "tok_per_s": 3689, "chain_tokens_median": 2009.0, "chain_tokens_p90": 6626.7, "chain_tokens_max": 32667, "parse_rate": 0.9889821615949633, "looped": 84, "minutes_to_deadline": 31.0}
(MainProcess pid=35900) INFO 10-31 05:10:50 [loggers.py:335] Engine 000: Avg prompt throughput: 496.6 tokens/s, Avg generation throughput: 3718.2 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 40.4%
progress {"time": "05:11:50", "elapsed_min": 112.0, "submitted": 2178, "finished": 1922, "in_flight": 256, "gen_tokens": 24874085, "tok_per_s": 3701, "chain_tokens_median": 2020.0, "chain_tokens_p90": 6659.6, "chain_tokens_max": 32667, "parse_rate": 0.9888137356919875, "looped": 86, "minutes_to_deadline": 30.0}
(MainProcess pid=35900) INFO 10-31 05:11:50 [loggers.py:335] Engine 000: Avg prompt throughput: 374.9 tokens/s, Avg generation throughput: 3867.5 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 83.7%, Prefix cache hit rate: 40.5%
progress {"time": "05:12:50", "elapsed_min": 113.0, "submitted": 2199, "finished": 1943, "in_flight": 256, "gen_tokens": 25114720, "tok_per_s": 3704, "chain_tokens_median": 2025.0, "chain_tokens_p90": 6655.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889346371590324, "looped": 86, "minutes_to_deadline": 29.0}
(MainProcess pid=35900) INFO 10-31 05:12:50 [loggers.py:335] Engine 000: Avg prompt throughput: 457.7 tokens/s, Avg generation throughput: 3914.7 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 40.6%
progress {"time": "05:13:50", "elapsed_min": 114.0, "submitted": 2221, "finished": 1965, "in_flight": 256, "gen_tokens": 25356254, "tok_per_s": 3707, "chain_tokens_median": 2013.5, "chain_tokens_p90": 6654.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889312977099237, "looped": 87, "minutes_to_deadline": 28.0}
(MainProcess pid=35900) INFO 10-31 05:13:50 [loggers.py:335] Engine 000: Avg prompt throughput: 387.8 tokens/s, Avg generation throughput: 3919.0 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 86.1%, Prefix cache hit rate: 40.4%
progress {"time": "05:14:50", "elapsed_min": 115.0, "submitted": 2239, "finished": 1983, "in_flight": 256, "gen_tokens": 25570370, "tok_per_s": 3705, "chain_tokens_median": 1999.5, "chain_tokens_p90": 6655.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.988905698436712, "looped": 88, "minutes_to_deadline": 27.0}
(MainProcess pid=35900) INFO 10-31 05:14:50 [loggers.py:335] Engine 000: Avg prompt throughput: 367.5 tokens/s, Avg generation throughput: 3871.7 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 86.6%, Prefix cache hit rate: 41.3%
progress {"time": "05:15:50", "elapsed_min": 116.0, "submitted": 2255, "finished": 1999, "in_flight": 256, "gen_tokens": 25836882, "tok_per_s": 3712, "chain_tokens_median": 2001.0, "chain_tokens_p90": 6658.0, "chain_tokens_max": 32667, "parse_rate": 0.9886193096548274, "looped": 91, "minutes_to_deadline": 26.0}
(MainProcess pid=35900) INFO 10-31 05:15:50 [loggers.py:335] Engine 000: Avg prompt throughput: 307.6 tokens/s, Avg generation throughput: 3923.5 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 85.3%, Prefix cache hit rate: 40.8%
progress {"time": "05:16:50", "elapsed_min": 117.0, "submitted": 2274, "finished": 2018, "in_flight": 256, "gen_tokens": 26057641, "tok_per_s": 3711, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6662.800000000001, "chain_tokens_max": 32667, "parse_rate": 0.9887264618434093, "looped": 91, "minutes_to_deadline": 25.0}
(MainProcess pid=35900) INFO 10-31 05:16:50 [loggers.py:335] Engine 000: Avg prompt throughput: 315.4 tokens/s, Avg generation throughput: 3962.0 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 86.7%, Prefix cache hit rate: 41.4%
progress {"time": "05:17:50", "elapsed_min": 118.0, "submitted": 2287, "finished": 2031, "in_flight": 256, "gen_tokens": 26254328, "tok_per_s": 3708, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6665.099999999999, "chain_tokens_max": 32667, "parse_rate": 0.9885524372230429, "looped": 93, "minutes_to_deadline": 24.0}
(MainProcess pid=35900) INFO 10-31 05:17:50 [loggers.py:335] Engine 000: Avg prompt throughput: 344.3 tokens/s, Avg generation throughput: 3953.5 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 39.9%
progress {"time": "05:18:50", "elapsed_min": 119.0, "submitted": 2301, "finished": 2045, "in_flight": 256, "gen_tokens": 26368518, "tok_per_s": 3693, "chain_tokens_median": 1991.5, "chain_tokens_p90": 6657.200000000001, "chain_tokens_max": 32667, "parse_rate": 0.9886308068459657, "looped": 93, "minutes_to_deadline": 23.0}
(MainProcess pid=35900) INFO 10-31 05:18:50 [loggers.py:335] Engine 000: Avg prompt throughput: 347.2 tokens/s, Avg generation throughput: 3880.8 tokens/s, Running: 256 reqs, Waiting: 607 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 40.2%
progress {"time": "05:19:50", "elapsed_min": 120.0, "submitted": 2324, "finished": 2068, "in_flight": 256, "gen_tokens": 26790397, "tok_per_s": 3720, "chain_tokens_median": 2010.0, "chain_tokens_p90": 6668.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9885154738878144, "looped": 95, "minutes_to_deadline": 22.0}
(MainProcess pid=35900) INFO 10-31 05:19:50 [loggers.py:335] Engine 000: Avg prompt throughput: 438.8 tokens/s, Avg generation throughput: 3892.9 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 40.5%
progress {"time": "05:20:50", "elapsed_min": 121.0, "submitted": 2348, "finished": 2092, "in_flight": 256, "gen_tokens": 27047980, "tok_per_s": 3725, "chain_tokens_median": 2012.5, "chain_tokens_p90": 6663.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9886472275334608, "looped": 95, "minutes_to_deadline": 21.0}
(MainProcess pid=35900) INFO 10-31 05:20:50 [loggers.py:335] Engine 000: Avg prompt throughput: 572.7 tokens/s, Avg generation throughput: 3999.0 tokens/s, Running: 255 reqs, Waiting: 619 reqs, GPU KV cache usage: 77.1%, Prefix cache hit rate: 41.4%
progress {"time": "05:21:50", "elapsed_min": 122.0, "submitted": 2373, "finished": 2117, "in_flight": 256, "gen_tokens": 27341202, "tok_per_s": 3735, "chain_tokens_median": 2012.5, "chain_tokens_p90": 6661.0, "chain_tokens_max": 32667, "parse_rate": 0.9887812942843647, "looped": 95, "minutes_to_deadline": 20.0}
(MainProcess pid=35900) INFO 10-31 05:21:50 [loggers.py:335] Engine 000: Avg prompt throughput: 541.2 tokens/s, Avg generation throughput: 4040.6 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 76.7%, Prefix cache hit rate: 41.0%
progress {"time": "05:22:50", "elapsed_min": 123.0, "submitted": 2393, "finished": 2137, "in_flight": 256, "gen_tokens": 27547550, "tok_per_s": 3732, "chain_tokens_median": 2002.5, "chain_tokens_p90": 6659.6, "chain_tokens_max": 32667, "parse_rate": 0.988886289190454, "looped": 95, "minutes_to_deadline": 19.0}
(MainProcess pid=35900) INFO 10-31 05:22:50 [loggers.py:335] Engine 000: Avg prompt throughput: 374.7 tokens/s, Avg generation throughput: 4123.5 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 79.3%, Prefix cache hit rate: 40.8%
progress {"time": "05:23:50", "elapsed_min": 124.0, "submitted": 2414, "finished": 2158, "in_flight": 256, "gen_tokens": 27769738, "tok_per_s": 3732, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6656.900000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889944392956441, "looped": 95, "minutes_to_deadline": 18.0}
(MainProcess pid=35900) INFO 10-31 05:23:50 [loggers.py:335] Engine 000: Avg prompt throughput: 457.2 tokens/s, Avg generation throughput: 3940.0 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 80.1%, Prefix cache hit rate: 41.0%
progress {"time": "05:24:50", "elapsed_min": 125.0, "submitted": 2439, "finished": 2183, "in_flight": 256, "gen_tokens": 27992542, "tok_per_s": 3732, "chain_tokens_median": 1974.0, "chain_tokens_p90": 6629.700000000002, "chain_tokens_max": 32667, "parse_rate": 0.98900595510765, "looped": 96, "minutes_to_deadline": 17.0}
(MainProcess pid=35900) INFO 10-31 05:24:50 [loggers.py:335] Engine 000: Avg prompt throughput: 302.6 tokens/s, Avg generation throughput: 4033.4 tokens/s, Running: 256 reqs, Waiting: 649 reqs, GPU KV cache usage: 80.5%, Prefix cache hit rate: 40.7%
progress {"time": "05:25:50", "elapsed_min": 126.0, "submitted": 2457, "finished": 2201, "in_flight": 256, "gen_tokens": 28182219, "tok_per_s": 3727, "chain_tokens_median": 1962.5, "chain_tokens_p90": 6626.7, "chain_tokens_max": 32667, "parse_rate": 0.988982280781463, "looped": 97, "minutes_to_deadline": 16.0}
(MainProcess pid=35900) INFO 10-31 05:25:50 [loggers.py:335] Engine 000: Avg prompt throughput: 450.7 tokens/s, Avg generation throughput: 3986.7 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 82.5%, Prefix cache hit rate: 40.6%
progress {"time": "05:26:50", "elapsed_min": 127.0, "submitted": 2476, "finished": 2220, "in_flight": 256, "gen_tokens": 28408410, "tok_per_s": 3728, "chain_tokens_median": 1957.5, "chain_tokens_p90": 6626.0, "chain_tokens_max": 32667, "parse_rate": 0.9889639639639639, "looped": 98, "minutes_to_deadline": 15.0}
(MainProcess pid=35900) INFO 10-31 05:26:50 [loggers.py:335] Engine 000: Avg prompt throughput: 426.8 tokens/s, Avg generation throughput: 3948.6 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 82.6%, Prefix cache hit rate: 41.0%
progress {"time": "05:27:50", "elapsed_min": 128.0, "submitted": 2495, "finished": 2239, "in_flight": 256, "gen_tokens": 28643720, "tok_per_s": 3729, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6626.5, "chain_tokens_max": 32667, "parse_rate": 0.9890576150066994, "looped": 98, "minutes_to_deadline": 14.0}
(MainProcess pid=35900) INFO 10-31 05:27:50 [loggers.py:335] Engine 000: Avg prompt throughput: 384.8 tokens/s, Avg generation throughput: 3897.1 tokens/s, Running: 256 reqs, Waiting: 628 reqs, GPU KV cache usage: 81.1%, Prefix cache hit rate: 39.8%
progress {"time": "05:28:50", "elapsed_min": 129.0, "submitted": 2516, "finished": 2260, "in_flight": 256, "gen_tokens": 28989670, "tok_per_s": 3745, "chain_tokens_median": 1959.0, "chain_tokens_p90": 6626.1, "chain_tokens_max": 32667, "parse_rate": 0.9889380530973452, "looped": 100, "minutes_to_deadline": 13.0}
(MainProcess pid=35900) INFO 10-31 05:28:50 [loggers.py:335] Engine 000: Avg prompt throughput: 338.9 tokens/s, Avg generation throughput: 4068.6 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 39.8%
progress {"time": "05:29:50", "elapsed_min": 130.0, "submitted": 2537, "finished": 2281, "in_flight": 256, "gen_tokens": 29280613, "tok_per_s": 3753, "chain_tokens_median": 1966.0, "chain_tokens_p90": 6627.0, "chain_tokens_max": 32667, "parse_rate": 0.9889302937308199, "looped": 101, "minutes_to_deadline": 12.0}
(MainProcess pid=35900) INFO 10-31 05:29:50 [loggers.py:335] Engine 000: Avg prompt throughput: 461.6 tokens/s, Avg generation throughput: 3987.0 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 75.9%, Prefix cache hit rate: 39.9%
progress {"time": "05:30:50", "elapsed_min": 131.0, "submitted": 2561, "finished": 2305, "in_flight": 256, "gen_tokens": 29568331, "tok_per_s": 3761, "chain_tokens_median": 1968.0, "chain_tokens_p90": 6627.0, "chain_tokens_max": 32667, "parse_rate": 0.9890455531453363, "looped": 101, "minutes_to_deadline": 11.0}
(MainProcess pid=35900) INFO 10-31 05:30:50 [loggers.py:335] Engine 000: Avg prompt throughput: 546.2 tokens/s, Avg generation throughput: 4038.2 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 40.8%
progress {"time": "05:31:50", "elapsed_min": 132.0, "submitted": 2587, "finished": 2331, "in_flight": 256, "gen_tokens": 29895411, "tok_per_s": 3774, "chain_tokens_median": 1968.0, "chain_tokens_p90": 6619.700000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889532389532389, "looped": 103, "minutes_to_deadline": 10.0}
(MainProcess pid=35900) INFO 10-31 05:31:50 [loggers.py:335] Engine 000: Avg prompt throughput: 515.7 tokens/s, Avg generation throughput: 4007.8 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 40.7%
progress {"time": "05:32:50", "elapsed_min": 133.0, "submitted": 2606, "finished": 2350, "in_flight": 256, "gen_tokens": 30078218, "tok_per_s": 3769, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6602.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9889361702127659, "looped": 104, "minutes_to_deadline": 9.0}
(MainProcess pid=35900) INFO 10-31 05:32:50 [loggers.py:335] Engine 000: Avg prompt throughput: 362.3 tokens/s, Avg generation throughput: 4367.1 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 74.7%, Prefix cache hit rate: 39.5%
progress {"time": "05:33:50", "elapsed_min": 134.0, "submitted": 2627, "finished": 2371, "in_flight": 256, "gen_tokens": 30315090, "tok_per_s": 3770, "chain_tokens_median": 1961.5, "chain_tokens_p90": 6584.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9890341628005062, "looped": 104, "minutes_to_deadline": 8.0}
(MainProcess pid=35900) INFO 10-31 05:33:50 [loggers.py:335] Engine 000: Avg prompt throughput: 324.7 tokens/s, Avg generation throughput: 4375.8 tokens/s, Running: 256 reqs, Waiting: 663 reqs, GPU KV cache usage: 77.1%, Prefix cache hit rate: 40.4%
progress {"time": "05:34:50", "elapsed_min": 135.0, "submitted": 2637, "finished": 2381, "in_flight": 256, "gen_tokens": 30456764, "tok_per_s": 3760, "chain_tokens_median": 1970.5, "chain_tokens_p90": 6584.4000000000015, "chain_tokens_max": 32667, "parse_rate": 0.9890802183956321, "looped": 104, "minutes_to_deadline": 7.0}
(MainProcess pid=35900) INFO 10-31 05:34:50 [loggers.py:335] Engine 000: Avg prompt throughput: 284.2 tokens/s, Avg generation throughput: 4265.1 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 81.6%, Prefix cache hit rate: 39.4%
progress {"time": "05:35:50", "elapsed_min": 136.0, "submitted": 2656, "finished": 2400, "in_flight": 256, "gen_tokens": 30709556, "tok_per_s": 3763, "chain_tokens_median": 1977.5, "chain_tokens_p90": 6585.0, "chain_tokens_max": 32667, "parse_rate": 0.9891666666666666, "looped": 104, "minutes_to_deadline": 6.0}
(MainProcess pid=35900) INFO 10-31 05:35:50 [loggers.py:335] Engine 000: Avg prompt throughput: 292.4 tokens/s, Avg generation throughput: 4149.7 tokens/s, Running: 256 reqs, Waiting: 662 reqs, GPU KV cache usage: 83.7%, Prefix cache hit rate: 39.8%
progress {"time": "05:36:50", "elapsed_min": 137.0, "submitted": 2669, "finished": 2413, "in_flight": 256, "gen_tokens": 30854327, "tok_per_s": 3753, "chain_tokens_median": 1981.5, "chain_tokens_p90": 6569.9, "chain_tokens_max": 32667, "parse_rate": 0.9892250310816411, "looped": 104, "minutes_to_deadline": 5.0}
(MainProcess pid=35900) INFO 10-31 05:36:50 [loggers.py:335] Engine 000: Avg prompt throughput: 382.3 tokens/s, Avg generation throughput: 4013.1 tokens/s, Running: 256 reqs, Waiting: 647 reqs, GPU KV cache usage: 85.1%, Prefix cache hit rate: 38.7%
progress {"time": "05:37:50", "elapsed_min": 138.0, "submitted": 2686, "finished": 2430, "in_flight": 256, "gen_tokens": 31070233, "tok_per_s": 3752, "chain_tokens_median": 1989.0, "chain_tokens_p90": 6568.1, "chain_tokens_max": 32667, "parse_rate": 0.9893004115226337, "looped": 104, "minutes_to_deadline": 4.0}
(MainProcess pid=35900) INFO 10-31 05:37:50 [loggers.py:335] Engine 000: Avg prompt throughput: 404.9 tokens/s, Avg generation throughput: 3885.1 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 40.2%
progress {"time": "05:38:50", "elapsed_min": 139.0, "submitted": 2702, "finished": 2446, "in_flight": 256, "gen_tokens": 31238513, "tok_per_s": 3745, "chain_tokens_median": 1986.0, "chain_tokens_p90": 6551.700000000001, "chain_tokens_max": 32667, "parse_rate": 0.9893704006541292, "looped": 104, "minutes_to_deadline": 3.0}
(MainProcess pid=35900) INFO 10-31 05:38:50 [loggers.py:335] Engine 000: Avg prompt throughput: 397.4 tokens/s, Avg generation throughput: 3927.2 tokens/s, Running: 256 reqs, Waiting: 634 reqs, GPU KV cache usage: 85.1%, Prefix cache hit rate: 41.3%
progress {"time": "05:39:50", "elapsed_min": 140.0, "submitted": 2717, "finished": 2461, "in_flight": 256, "gen_tokens": 31464710, "tok_per_s": 3745, "chain_tokens_median": 1993.5, "chain_tokens_p90": 6554.100000000002, "chain_tokens_max": 32667, "parse_rate": 0.9894351889475823, "looped": 104, "minutes_to_deadline": 2.0}
(MainProcess pid=35900) INFO 10-31 05:39:50 [loggers.py:335] Engine 000: Avg prompt throughput: 408.7 tokens/s, Avg generation throughput: 3884.8 tokens/s, Running: 256 reqs, Waiting: 616 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 40.6%
progress {"time": "05:40:50", "elapsed_min": 141.0, "submitted": 2741, "finished": 2485, "in_flight": 256, "gen_tokens": 31833728, "tok_per_s": 3762, "chain_tokens_median": 1991.5, "chain_tokens_p90": 6570.700000000003, "chain_tokens_max": 32667, "parse_rate": 0.9892354124748491, "looped": 107, "minutes_to_deadline": 1.0}
(MainProcess pid=35900) INFO 10-31 05:40:50 [loggers.py:335] Engine 000: Avg prompt throughput: 385.7 tokens/s, Avg generation throughput: 3961.8 tokens/s, Running: 255 reqs, Waiting: 641 reqs, GPU KV cache usage: 81.1%, Prefix cache hit rate: 40.8%
progress {"time": "05:41:50", "elapsed_min": 142.0, "submitted": 2757, "finished": 2501, "in_flight": 256, "gen_tokens": 32068955, "tok_per_s": 3763, "chain_tokens_median": 1999.0, "chain_tokens_p90": 6574.900000000005, "chain_tokens_max": 32667, "parse_rate": 0.989204318272691, "looped": 108, "minutes_to_deadline": -0.0}
(MainProcess pid=35900) INFO 10-31 05:41:50 [loggers.py:335] Engine 000: Avg prompt throughput: 352.1 tokens/s, Avg generation throughput: 4021.7 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 82.5%, Prefix cache hit rate: 41.2%
progress {"time": "05:42:50", "elapsed_min": 143.0, "submitted": 2757, "finished": 2517, "in_flight": 240, "gen_tokens": 32229386, "tok_per_s": 3756, "chain_tokens_median": 1991.5, "chain_tokens_p90": 6569.300000000001, "chain_tokens_max": 32667, "parse_rate": 0.9892729439809297, "looped": 108, "minutes_to_deadline": -1.0}
(MainProcess pid=35900) INFO 10-31 05:42:50 [loggers.py:335] Engine 000: Avg prompt throughput: 323.1 tokens/s, Avg generation throughput: 3953.2 tokens/s, Running: 256 reqs, Waiting: 559 reqs, GPU KV cache usage: 83.5%, Prefix cache hit rate: 41.2%
progress {"time": "05:43:50", "elapsed_min": 144.0, "submitted": 2757, "finished": 2548, "in_flight": 209, "gen_tokens": 32584742, "tok_per_s": 3771, "chain_tokens_median": 1978.5, "chain_tokens_p90": 6568.9, "chain_tokens_max": 32667, "parse_rate": 0.9892072213500785, "looped": 110, "minutes_to_deadline": -2.0}
(MainProcess pid=35900) INFO 10-31 05:43:50 [loggers.py:335] Engine 000: Avg prompt throughput: 409.0 tokens/s, Avg generation throughput: 3901.6 tokens/s, Running: 256 reqs, Waiting: 467 reqs, GPU KV cache usage: 80.1%, Prefix cache hit rate: 41.1%
progress {"time": "05:44:50", "elapsed_min": 145.0, "submitted": 2757, "finished": 2564, "in_flight": 193, "gen_tokens": 32749380, "tok_per_s": 3764, "chain_tokens_median": 1973.0, "chain_tokens_p90": 6547.0, "chain_tokens_max": 32667, "parse_rate": 0.9891770670826833, "looped": 111, "minutes_to_deadline": -3.0}
(MainProcess pid=35900) INFO 10-31 05:44:50 [loggers.py:335] Engine 000: Avg prompt throughput: 429.9 tokens/s, Avg generation throughput: 3944.5 tokens/s, Running: 256 reqs, Waiting: 387 reqs, GPU KV cache usage: 81.7%, Prefix cache hit rate: 41.2%
progress {"time": "05:45:51", "elapsed_min": 146.0, "submitted": 2757, "finished": 2581, "in_flight": 176, "gen_tokens": 32954949, "tok_per_s": 3761, "chain_tokens_median": 1966.0, "chain_tokens_p90": 6554.100000000002, "chain_tokens_max": 32667, "parse_rate": 0.9891514916698954, "looped": 112, "minutes_to_deadline": -4.0}
(MainProcess pid=35900) INFO 10-31 05:45:51 [loggers.py:335] Engine 000: Avg prompt throughput: 336.4 tokens/s, Avg generation throughput: 4034.4 tokens/s, Running: 256 reqs, Waiting: 328 reqs, GPU KV cache usage: 83.9%, Prefix cache hit rate: 42.1%
progress {"time": "05:46:51", "elapsed_min": 147.0, "submitted": 2757, "finished": 2594, "in_flight": 163, "gen_tokens": 33088724, "tok_per_s": 3751, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6551.5, "chain_tokens_max": 32667, "parse_rate": 0.9891094834232845, "looped": 113, "minutes_to_deadline": -5.0}
(MainProcess pid=35900) INFO 10-31 05:46:51 [loggers.py:335] Engine 000: Avg prompt throughput: 277.8 tokens/s, Avg generation throughput: 4064.4 tokens/s, Running: 256 reqs, Waiting: 268 reqs, GPU KV cache usage: 87.5%, Prefix cache hit rate: 41.8%
progress {"time": "05:47:51", "elapsed_min": 148.0, "submitted": 2757, "finished": 2608, "in_flight": 149, "gen_tokens": 33216830, "tok_per_s": 3740, "chain_tokens_median": 1954.5, "chain_tokens_p90": 6538.699999999999, "chain_tokens_max": 32667, "parse_rate": 0.989167944785276, "looped": 113, "minutes_to_deadline": -6.0}
(MainProcess pid=35900) INFO 10-31 05:47:51 [loggers.py:335] Engine 000: Avg prompt throughput: 263.2 tokens/s, Avg generation throughput: 3970.5 tokens/s, Running: 256 reqs, Waiting: 213 reqs, GPU KV cache usage: 90.4%, Prefix cache hit rate: 41.9%
progress {"time": "05:48:51", "elapsed_min": 149.0, "submitted": 2757, "finished": 2623, "in_flight": 134, "gen_tokens": 33414248, "tok_per_s": 3737, "chain_tokens_median": 1957.0, "chain_tokens_p90": 6541.699999999999, "chain_tokens_max": 32667, "parse_rate": 0.989229889439573, "looped": 113, "minutes_to_deadline": -7.0}
(MainProcess pid=35900) INFO 10-31 05:48:51 [loggers.py:335] Engine 000: Avg prompt throughput: 218.9 tokens/s, Avg generation throughput: 3932.4 tokens/s, Running: 256 reqs, Waiting: 161 reqs, GPU KV cache usage: 94.1%, Prefix cache hit rate: 41.4%
progress {"time": "05:49:51", "elapsed_min": 150.0, "submitted": 2757, "finished": 2636, "in_flight": 121, "gen_tokens": 33635716, "tok_per_s": 3737, "chain_tokens_median": 1962.5, "chain_tokens_p90": 6565.000000000007, "chain_tokens_max": 32667, "parse_rate": 0.9892830045523521, "looped": 113, "minutes_to_deadline": -8.0}
(MainProcess pid=35900) INFO 10-31 05:49:51 [loggers.py:335] Engine 000: Avg prompt throughput: 325.9 tokens/s, Avg generation throughput: 3774.2 tokens/s, Running: 255 reqs, Waiting: 94 reqs, GPU KV cache usage: 92.3%, Prefix cache hit rate: 41.4%
progress {"time": "05:50:51", "elapsed_min": 151.0, "submitted": 2757, "finished": 2652, "in_flight": 105, "gen_tokens": 33814899, "tok_per_s": 3732, "chain_tokens_median": 1967.0, "chain_tokens_p90": 6552.900000000003, "chain_tokens_max": 32667, "parse_rate": 0.9893476621417798, "looped": 113, "minutes_to_deadline": -9.0}
(MainProcess pid=35900) INFO 10-31 05:50:51 [loggers.py:335] Engine 000: Avg prompt throughput: 375.4 tokens/s, Avg generation throughput: 3710.2 tokens/s, Running: 256 reqs, Waiting: 15 reqs, GPU KV cache usage: 90.8%, Prefix cache hit rate: 41.7%
progress {"time": "05:51:51", "elapsed_min": 152.0, "submitted": 2757, "finished": 2675, "in_flight": 82, "gen_tokens": 34201661, "tok_per_s": 3750, "chain_tokens_median": 1968.0, "chain_tokens_p90": 6593.1, "chain_tokens_max": 32667, "parse_rate": 0.9893457943925233, "looped": 114, "minutes_to_deadline": -10.0}
(MainProcess pid=35900) INFO 10-31 05:51:51 [loggers.py:335] Engine 000: Avg prompt throughput: 72.4 tokens/s, Avg generation throughput: 3792.0 tokens/s, Running: 179 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 41.1%
progress {"time": "05:52:51", "elapsed_min": 153.0, "submitted": 2757, "finished": 2691, "in_flight": 66, "gen_tokens": 34439537, "tok_per_s": 3751, "chain_tokens_median": 1978.5, "chain_tokens_p90": 6604.800000000003, "chain_tokens_max": 32667, "parse_rate": 0.9894091415830546, "looped": 114, "minutes_to_deadline": -11.0}
(MainProcess pid=35900) INFO 10-31 05:52:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3143.7 tokens/s, Running: 133 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.7%, Prefix cache hit rate: 41.1%
progress {"time": "05:53:51", "elapsed_min": 154.0, "submitted": 2757, "finished": 2706, "in_flight": 51, "gen_tokens": 34801910, "tok_per_s": 3766, "chain_tokens_median": 1996.5, "chain_tokens_p90": 6629.100000000002, "chain_tokens_max": 32667, "parse_rate": 0.989190687361419, "looped": 117, "minutes_to_deadline": -12.0}
(MainProcess pid=35900) INFO 10-31 05:53:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2680.2 tokens/s, Running: 94 reqs, Waiting: 0 reqs, GPU KV cache usage: 58.4%, Prefix cache hit rate: 41.1%
progress {"time": "05:54:51", "elapsed_min": 155.0, "submitted": 2757, "finished": 2718, "in_flight": 39, "gen_tokens": 35051862, "tok_per_s": 3769, "chain_tokens_median": 2011.5, "chain_tokens_p90": 6649.699999999999, "chain_tokens_max": 32667, "parse_rate": 0.9891464311994114, "looped": 118, "minutes_to_deadline": -13.0}
(MainProcess pid=35900) INFO 10-31 05:54:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2183.3 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 51.6%, Prefix cache hit rate: 41.1%
progress {"time": "05:55:51", "elapsed_min": 156.0, "submitted": 2757, "finished": 2724, "in_flight": 33, "gen_tokens": 35211479, "tok_per_s": 3761, "chain_tokens_median": 2018.5, "chain_tokens_p90": 6660.0, "chain_tokens_max": 32667, "parse_rate": 0.9890785609397944, "looped": 119, "minutes_to_deadline": -14.0}
(MainProcess pid=35900) INFO 10-31 05:55:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1754.1 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 43.0%, Prefix cache hit rate: 41.1%
progress {"time": "05:56:51", "elapsed_min": 157.0, "submitted": 2757, "finished": 2732, "in_flight": 25, "gen_tokens": 35544005, "tok_per_s": 3773, "chain_tokens_median": 2027.0, "chain_tokens_p90": 6684.300000000001, "chain_tokens_max": 32667, "parse_rate": 0.9889275256222547, "looped": 121, "minutes_to_deadline": -15.0}
(MainProcess pid=35900) INFO 10-31 05:56:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1510.1 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.0%, Prefix cache hit rate: 41.1%
progress {"time": "05:57:51", "elapsed_min": 158.0, "submitted": 2757, "finished": 2737, "in_flight": 20, "gen_tokens": 35783010, "tok_per_s": 3774, "chain_tokens_median": 2033.5, "chain_tokens_p90": 6719.600000000002, "chain_tokens_max": 32667, "parse_rate": 0.9887650712458896, "looped": 123, "minutes_to_deadline": -16.0}
(MainProcess pid=35900) INFO 10-31 05:57:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1310.6 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.2%, Prefix cache hit rate: 41.1%
progress {"time": "05:58:51", "elapsed_min": 159.0, "submitted": 2757, "finished": 2738, "in_flight": 19, "gen_tokens": 35836233, "tok_per_s": 3756, "chain_tokens_median": 2035.0, "chain_tokens_p90": 6722.9, "chain_tokens_max": 32667, "parse_rate": 0.9886778670562454, "looped": 124, "minutes_to_deadline": -17.0}
(MainProcess pid=35900) INFO 10-31 05:58:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1234.4 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.2%, Prefix cache hit rate: 41.1%
progress {"time": "05:59:51", "elapsed_min": 160.0, "submitted": 2757, "finished": 2746, "in_flight": 11, "gen_tokens": 36307083, "tok_per_s": 3781, "chain_tokens_median": 2049.5, "chain_tokens_p90": 6773.0, "chain_tokens_max": 32667, "parse_rate": 0.98761835396941, "looped": 136, "minutes_to_deadline": -18.0}
(MainProcess pid=35900) INFO 10-31 05:59:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1096.8 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.2%, Prefix cache hit rate: 41.1%
progress {"time": "06:00:51", "elapsed_min": 161.0, "submitted": 2757, "finished": 2747, "in_flight": 10, "gen_tokens": 36364540, "tok_per_s": 3764, "chain_tokens_median": 2051.5, "chain_tokens_p90": 6777.70000000001, "chain_tokens_max": 32667, "parse_rate": 0.9875318529304696, "looped": 137, "minutes_to_deadline": -19.0}
(MainProcess pid=35900) INFO 10-31 06:00:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 892.5 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 41.1%
progress {"time": "06:01:51", "elapsed_min": 162.0, "submitted": 2757, "finished": 2750, "in_flight": 7, "gen_tokens": 36515259, "tok_per_s": 3756, "chain_tokens_median": 2056.5, "chain_tokens_p90": 6791.1, "chain_tokens_max": 32667, "parse_rate": 0.9872727272727273, "looped": 140, "minutes_to_deadline": -20.0}
(MainProcess pid=35900) INFO 10-31 06:01:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 716.7 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 41.1%
progress {"time": "06:02:51", "elapsed_min": 163.0, "submitted": 2757, "finished": 2754, "in_flight": 3, "gen_tokens": 36697448, "tok_per_s": 3752, "chain_tokens_median": 2058.5, "chain_tokens_p90": 6802.5, "chain_tokens_max": 32667, "parse_rate": 0.9869281045751634, "looped": 144, "minutes_to_deadline": -21.0}
(MainProcess pid=35900) INFO 10-31 06:02:51 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 622.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 41.1%
(MainProcess pid=35900) INFO 10-31 06:03:04 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=35900) INFO 10-31 06:03:04 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=35900) INFO 10-31 06:03:04 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=36321) INFO 10-31 06:03:04 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=36321) INFO 10-31 06:03:04 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=36321) INFO 10-31 06:03:04 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=36321) INFO 10-31 06:03:04 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=35900) WARNING 10-31 06:03:09 [utils.py:648] [shutdown] Process manager: force killing remaining processes count=1
(MainProcess pid=35900) WARNING 10-31 06:03:09 [utils.py:653] [shutdown] Process manager: force killing remaining process EngineCore pid 36321
(MainProcess pid=35900) INFO 10-31 06:03:09 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=35900) INFO 10-31 06:03:09 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=35900) INFO 10-31 06:03:09 [core_client.py:777] [shutdown] MPClient: complete
{
"items": 3169,
"samples": 12676,
"tokens_total": 40974033,
"tokens_median": 1945.0,
"tokens_p90": 6661.0,
"tokens_max": 32667,
"parse_rate": 0.9878510571158094,
"looped": 154,
"items_with_correct_chain": 2818,
"majority_accuracy": 0.8520037866834964,
"this_run": {
"submitted": 2757,
"finished": 2757,
"gen_tokens": 36922408,
"t0": 1793416789.8045087,
"seconds": 9795.067398786545,
"tok_per_s": 3769.489937820551
},
"seconds_this_run": 10013.39470243454
}
{"time": "2026-10-31T06:03:15Z", "event": "done", "stage": "generate", "hours_left": 3.44}
{"time": "2026-10-31T06:03:35Z", "event": "start", "stage": "train_joint", "hours_left": 3.44}
{"examples": 5987, "tokens": 9210112, "readout_from_votes": 3162, "reasoner_examples": 2818, "reasoner_no_correct_chain": 351, "readout_gold_fallback": 7}
Loading weights: 0%| | 0/426 [00:00<?, ?it/s]Loading weights: 85%|████████▍ | 361/426 [00:00<00:00, 3587.39it/s]Loading weights: 100%|██████████| 426/426 [00:00<00:00, 3588.38it/s]
trainable params: 60,948,480 || all params: 4,266,699,776 || trainable%: 1.4285
[transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
/workspace/decision-reasoner/train.py:158: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
Consider using tensor.detach() first. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:822.)
stats["chain_loss"] += float(loss)
{"probe_tok_per_s": 2110.0069727294785, "fit_tokens": 8728120.773889866, "have_tokens": 9210112}
{"kept_fraction": 0.9476671699421099, "examples_after_drop": 5669}
{"step": 1, "of": 207, "lr": 1.6666666666666667e-05, "grad_norm": 16.353824615478516, "examples": 30, "readout_examples": 14, "reasoner_examples": 16, "step_tokens": 40439, "chain_tokens": 21065, "micro_batches": 3, "tok_per_s": 2929.685804489287, "elapsed_min": 0.23005311091740926, "eta_min": 47.39094084898631, "chain_loss": 0.1679091975092888, "readout_loss": 1.934458957186767, "readout_argmax_matches_target": 0.42857142857142855, "readout_mean_top_prob": 0.5294180842382568, "readout_target_entropy": 0.2086534925869533, "mem_gb": 19.085318656}
{"step": 2, "of": 207, "lr": 3.3333333333333335e-05, "grad_norm": 7.542080402374268, "examples": 8, "readout_examples": 1, "reasoner_examples": 7, "step_tokens": 43052, "chain_tokens": 26243, "micro_batches": 3, "tok_per_s": 2991.560821974237, "elapsed_min": 0.46514737606048584, "eta_min": 47.6776060461998, "chain_loss": 0.22998714393803052, "readout_loss": 0.6888041496276855, "readout_argmax_matches_target": 1.0, "readout_mean_top_prob": 0.5021762847900391, "readout_target_entropy": 0.0, "mem_gb": 19.085318656}
{"step": 3, "of": 207, "lr": 5e-05, "grad_norm": 12.8873872756958, "examples": 61, "readout_examples": 57, "reasoner_examples": 4, "step_tokens": 39601, "chain_tokens": 22006, "micro_batches": 3, "tok_per_s": 1936.670188000899, "elapsed_min": 1.0593096057573954, "eta_min": 72.03305319150289, "chain_loss": 0.35129909589886665, "readout_loss": 0.8425638817338551, "readout_argmax_matches_target": 0.5964912280701754, "readout_mean_top_prob": 0.6768302318819782, "readout_target_entropy": 0.16120485222914763, "mem_gb": 19.085318656}
{"step": 10, "of": 207, "lr": 9.99023153874608e-05, "grad_norm": 6.824309349060059, "examples": 31, "readout_examples": 5, "reasoner_examples": 26, "step_tokens": 46378, "chain_tokens": 26181, "micro_batches": 3, "tok_per_s": 2601.1282333324484, "elapsed_min": 2.768458922704061, "eta_min": 54.53864077727, "chain_loss": 0.1670885752313412, "readout_loss": 0.9845329189673067, "readout_argmax_matches_target": 0.6, "readout_mean_top_prob": 0.6901487231254577, "readout_target_entropy": 0.10656597912311554, "mem_gb": 19.085318656}
{"step": 20, "of": 207, "lr": 9.880774227376726e-05, "grad_norm": 4.599339008331299, "examples": 28, "readout_examples": 9, "reasoner_examples": 19, "step_tokens": 40504, "chain_tokens": 30459, "micro_batches": 3, "tok_per_s": 2807.820395148787, "elapsed_min": 5.093814651171367, "eta_min": 47.627166988452274, "chain_loss": 0.21822420193960793, "readout_loss": 1.1004138191541035, "readout_argmax_matches_target": 0.5555555555555556, "readout_mean_top_prob": 0.5338592761092715, "readout_target_entropy": 0.0, "mem_gb": 19.085318656}
{"step": 30, "of": 207, "lr": 9.652326481535435e-05, "grad_norm": 12.56348991394043, "examples": 22, "readout_examples": 8, "reasoner_examples": 14, "step_tokens": 36999, "chain_tokens": 23412, "micro_batches": 3, "tok_per_s": 2961.0439312334515, "elapsed_min": 7.217820413907369, "eta_min": 42.58514044205348, "chain_loss": 0.18116038452301705, "readout_loss": 1.2184196175076067, "readout_argmax_matches_target": 0.625, "readout_mean_top_prob": 0.7747065685689449, "readout_target_entropy": 0.31010043574497104, "mem_gb": 19.085318656}
{"step": 40, "of": 207, "lr": 9.310457724282034e-05, "grad_norm": 2.214334011077881, "examples": 122, "readout_examples": 110, "reasoner_examples": 12, "step_tokens": 39960, "chain_tokens": 14950, "micro_batches": 3, "tok_per_s": 2968.213244752267, "elapsed_min": 9.658481709162395, "eta_min": 40.324161135752995, "chain_loss": 0.17671912411848703, "readout_loss": 0.6666781993836842, "readout_argmax_matches_target": 0.6909090909090909, "readout_mean_top_prob": 0.732171431183815, "readout_target_entropy": 0.22204227608807825, "mem_gb": 19.085318656}
{"step": 50, "of": 207, "lr": 8.863502516910058e-05, "grad_norm": 0.2128523737192154, "examples": 14, "readout_examples": 0, "reasoner_examples": 14, "step_tokens": 42258, "chain_tokens": 33152, "micro_batches": 3, "tok_per_s": 2991.9116578971434, "elapsed_min": 11.975854937235514, "eta_min": 37.60418450291952, "chain_loss": 0.15201139077544212, "readout_loss": null, "readout_argmax_matches_target": null, "readout_mean_top_prob": null, "readout_target_entropy": null, "mem_gb": 19.085318656}
{"step": 60, "of": 207, "lr": 8.322357367194109e-05, "grad_norm": 8.037028312683105, "examples": 30, "readout_examples": 12, "reasoner_examples": 18, "step_tokens": 44774, "chain_tokens": 30106, "micro_batches": 3, "tok_per_s": 3034.553312825412, "elapsed_min": 14.053682899475097, "eta_min": 34.431523103713985, "chain_loss": 0.22104241698980331, "readout_loss": 1.124840331884722, "readout_argmax_matches_target": 0.4166666666666667, "readout_mean_top_prob": 0.5562434966365496, "readout_target_entropy": 0.07488711923360825, "mem_gb": 19.085318656}
{"step": 70, "of": 207, "lr": 7.700215078873379e-05, "grad_norm": 6.59724235534668, "examples": 24, "readout_examples": 8, "reasoner_examples": 16, "step_tokens": 42708, "chain_tokens": 30189, "micro_batches": 3, "tok_per_s": 3048.597837583298, "elapsed_min": 16.3681587378184, "eta_min": 32.03482495830172, "chain_loss": 0.1460376912727952, "readout_loss": 0.8100452562794089, "readout_argmax_matches_target": 0.625, "readout_mean_top_prob": 0.6200514361262321, "readout_target_entropy": 0.17352180555462837, "mem_gb": 19.085318656}
{"step": 80, "of": 207, "lr": 7.012243118776269e-05, "grad_norm": 4.699362277984619, "examples": 13, "readout_examples": 1, "reasoner_examples": 12, "step_tokens": 42375, "chain_tokens": 31147, "micro_batches": 3, "tok_per_s": 3029.414900353085, "elapsed_min": 18.69640459616979, "eta_min": 29.680542296419542, "chain_loss": 0.20653490660091242, "readout_loss": 0.7964814901351929, "readout_argmax_matches_target": 1.0, "readout_mean_top_prob": 0.4509127140045166, "readout_target_entropy": 0.0, "mem_gb": 19.085318656}
{"step": 90, "of": 207, "lr": 6.275213842808383e-05, "grad_norm": 7.471736907958984, "examples": 15, "readout_examples": 3, "reasoner_examples": 12, "step_tokens": 44852, "chain_tokens": 27630, "micro_batches": 3, "tok_per_s": 3057.3651444803836, "elapsed_min": 20.847308162848154, "eta_min": 27.101500611702598, "chain_loss": 0.20712771515051523, "readout_loss": 1.5815004110336304, "readout_argmax_matches_target": 0.3333333333333333, "readout_mean_top_prob": 0.42134852210680646, "readout_target_entropy": 0.12559005618095398, "mem_gb": 19.085318656}
{"step": 100, "of": 207, "lr": 5.5070955956790594e-05, "grad_norm": 0.28775501251220703, "examples": 15, "readout_examples": 0, "reasoner_examples": 15, "step_tokens": 45791, "chain_tokens": 33489, "micro_batches": 3, "tok_per_s": 3079.1171606366233, "elapsed_min": 23.077578071753184, "eta_min": 24.693008536775906, "chain_loss": 0.18781028439601263, "readout_loss": null, "readout_argmax_matches_target": null, "readout_mean_top_prob": null, "readout_target_entropy": null, "mem_gb": 19.085318656}
{"step": 110, "of": 207, "lr": 4.726614653117071e-05, "grad_norm": 2.901421070098877, "examples": 64, "readout_examples": 60, "reasoner_examples": 4, "step_tokens": 40137, "chain_tokens": 21425, "micro_batches": 3, "tok_per_s": 3093.3938058169592, "elapsed_min": 25.262571867307027, "eta_min": 22.276995192079834, "chain_loss": 0.2232692390680313, "readout_loss": 0.7240575540345162, "readout_argmax_matches_target": 0.6666666666666666, "readout_mean_top_prob": 0.7004167820016544, "readout_target_entropy": 0.17394846379756929, "mem_gb": 19.085318656}
{"step": 120, "of": 207, "lr": 3.952798686169279e-05, "grad_norm": 0.2123325616121292, "examples": 14, "readout_examples": 0, "reasoner_examples": 14, "step_tokens": 43544, "chain_tokens": 33440, "micro_batches": 3, "tok_per_s": 3097.7480998700826, "elapsed_min": 27.570705314477284, "eta_min": 19.988761352996033, "chain_loss": 0.18893495627811976, "readout_loss": null, "readout_argmax_matches_target": null, "readout_mean_top_prob": null, "readout_target_entropy": null, "mem_gb": 19.085318656}
{"step": 130, "of": 207, "lr": 3.2045128776568784e-05, "grad_norm": 12.950605392456055, "examples": 71, "readout_examples": 65, "reasoner_examples": 6, "step_tokens": 39441, "chain_tokens": 20833, "micro_batches": 3, "tok_per_s": 3107.8540108273446, "elapsed_min": 29.71120361884435, "eta_min": 17.598174451161654, "chain_loss": 0.22970323637127876, "readout_loss": 0.7027810201084671, "readout_argmax_matches_target": 0.6615384615384615, "readout_mean_top_prob": 0.6789758677666003, "readout_target_entropy": 0.18075014352798463, "mem_gb": 19.085318656}
{"step": 140, "of": 207, "lr": 2.500000000000001e-05, "grad_norm": 1.572521686553955, "examples": 19, "readout_examples": 5, "reasoner_examples": 14, "step_tokens": 45279, "chain_tokens": 30104, "micro_batches": 3, "tok_per_s": 3116.8414919136612, "elapsed_min": 31.818113601207735, "eta_min": 15.227240080577985, "chain_loss": 0.22058590341891562, "readout_loss": 0.591015362739563, "readout_argmax_matches_target": 0.8, "readout_mean_top_prob": 0.6042369425296783, "readout_target_entropy": 0.0, "mem_gb": 19.085318656}
{"step": 150, "of": 207, "lr": 1.8564356670455767e-05, "grad_norm": 4.612676620483398, "examples": 15, "readout_examples": 2, "reasoner_examples": 13, "step_tokens": 46038, "chain_tokens": 31670, "micro_batches": 3, "tok_per_s": 3123.031223932227, "elapsed_min": 34.00748900175095, "eta_min": 12.92284582066536, "chain_loss": 0.17214029081738913, "readout_loss": 2.996444523334503, "readout_argmax_matches_target": 0.0, "readout_mean_top_prob": 0.713391900062561, "readout_target_entropy": 0.0, "mem_gb": 19.085318656}
{"step": 160, "of": 207, "lr": 1.289509602599996e-05, "grad_norm": 0.22141993045806885, "examples": 8, "readout_examples": 0, "reasoner_examples": 8, "step_tokens": 37780, "chain_tokens": 33914, "micro_batches": 3, "tok_per_s": 3139.090986909638, "elapsed_min": 36.10835126241048, "eta_min": 10.606828183333079, "chain_loss": 0.24487851280719042, "readout_loss": null, "readout_argmax_matches_target": null, "readout_mean_top_prob": null, "readout_target_entropy": null, "mem_gb": 19.085318656}
{"step": 170, "of": 207, "lr": 8.130431340959981e-06, "grad_norm": 0.2147686630487442, "examples": 10, "readout_examples": 0, "reasoner_examples": 10, "step_tokens": 36655, "chain_tokens": 27672, "micro_batches": 3, "tok_per_s": 3145.1973456729534, "elapsed_min": 38.23120463291804, "eta_min": 8.320909243635104, "chain_loss": 0.170370851829648, "readout_loss": null, "readout_argmax_matches_target": null, "readout_mean_top_prob": null, "readout_target_entropy": null, "mem_gb": 19.085318656}
{"step": 180, "of": 207, "lr": 4.386522366755169e-06, "grad_norm": 4.399493217468262, "examples": 22, "readout_examples": 9, "reasoner_examples": 13, "step_tokens": 46612, "chain_tokens": 32152, "micro_batches": 4, "tok_per_s": 3117.725601520395, "elapsed_min": 40.83136756420136, "eta_min": 6.124705134630203, "chain_loss": 0.19615883895984063, "readout_loss": 0.499316999481784, "readout_argmax_matches_target": 1.0, "readout_mean_top_prob": 0.6652993592951033, "readout_target_entropy": 0.15321971227725348, "mem_gb": 19.782436352}
{"step": 190, "of": 207, "lr": 1.7546434247760146e-06, "grad_norm": 0.7283493280410767, "examples": 17, "readout_examples": 2, "reasoner_examples": 15, "step_tokens": 44357, "chain_tokens": 28694, "micro_batches": 3, "tok_per_s": 3129.445585387388, "elapsed_min": 42.86470121939977, "eta_min": 3.835262740683137, "chain_loss": 0.15698070228099822, "readout_loss": 0.5103972256183624, "readout_argmax_matches_target": 1.0, "readout_mean_top_prob": 0.8205999135971069, "readout_target_entropy": 0.18838508427143097, "mem_gb": 19.782436352}
{"step": 200, "of": 207, "lr": 2.9895819156119943e-07, "grad_norm": 1.8945939540863037, "examples": 19, "readout_examples": 2, "reasoner_examples": 17, "step_tokens": 43523, "chain_tokens": 28779, "micro_batches": 3, "tok_per_s": 3134.7371156418453, "elapsed_min": 45.05890333255132, "eta_min": 1.5770616166392961, "chain_loss": 0.17900991264511557, "readout_loss": 0.46608100086450577, "readout_argmax_matches_target": 1.0, "readout_mean_top_prob": 0.6638901978731155, "readout_target_entropy": 0.0, "mem_gb": 19.782436352}
{"mode": "joint", "alpha": 0.5, "lr": 0.0001, "rank": 32, "lora_alpha": 64, "seed": 17, "examples": 5669, "steps_done": 207, "steps_planned": 207, "stopped_at_deadline": false, "train_tokens": 8743467, "train_seconds": 2787.2046253681183, "tok_per_s": 3137.0021850639014, "counts": {"readout_from_votes": 3162, "reasoner_examples": 2818, "reasoner_no_correct_chain": 351, "readout_gold_fallback": 7, "dropped_for_time": 318}, "reasoner_examples": 2673, "readout_examples": 2996}
{"time": "2026-10-31T06:54:16Z", "event": "done", "stage": "train_joint", "hours_left": 2.59}
{"time": "2026-10-31T06:54:39Z", "event": "start", "stage": "merge_joint", "hours_left": 2.59}
Loading weights: 0%| | 0/426 [00:00<?, ?it/s]Loading weights: 92%|█████████▏| 391/426 [00:00<00:00, 3909.51it/s]Loading weights: 100%|██████████| 426/426 [00:00<00:00, 3965.79it/s]
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]Writing model shards: 100%|██████████| 1/1 [00:03<00:00, 3.16s/it]Writing model shards: 100%|██████████| 1/1 [00:03<00:00, 3.16s/it]
merged results/adapters/joint into checkpoints/joint_merged
{"time": "2026-10-31T06:55:40Z", "event": "done", "stage": "merge_joint", "hours_left": 2.57}
{"time": "2026-10-31T06:55:59Z", "event": "start", "stage": "train_gold", "hours_left": 2.56}
{"examples": 3169, "tokens": 1711105, "readout_from_votes": 3162, "readout_gold_fallback": 7}
Loading weights: 0%| | 0/426 [00:00<?, ?it/s]Loading weights: 83%|████████▎ | 353/426 [00:00<00:00, 3528.88it/s]Loading weights: 100%|██████████| 426/426 [00:00<00:00, 3504.75it/s]
trainable params: 60,948,480 || all params: 4,266,699,776 || trainable%: 1.4285
[transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
/workspace/decision-reasoner/train.py:145: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
Consider using tensor.detach() first. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:822.)
stats["readout_loss"] += float(loss)
{"probe_tok_per_s": 3128.0360783164774, "fit_tokens": 4730778.434835311, "have_tokens": 1711105}
{"step": 1, "of": 37, "lr": 0.0001, "grad_norm": 17.038654327392578, "examples": 50, "readout_examples": 50, "reasoner_examples": 0, "step_tokens": 47327, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 3586.047278836919, "elapsed_min": 0.21995898882548015, "eta_min": 7.918523597717285, "chain_loss": null, "readout_loss": 1.3595495656877756, "readout_argmax_matches_target": 0.46, "readout_mean_top_prob": 0.5607822763919831, "readout_target_entropy": 0.0065016597509384155, "mem_gb": 17.69098752}
{"step": 2, "of": 37, "lr": 9.980973490458728e-05, "grad_norm": 19.448925018310547, "examples": 61, "readout_examples": 61, "reasoner_examples": 0, "step_tokens": 47314, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 3618.9956881822372, "elapsed_min": 0.43585296471913654, "eta_min": 7.627426882584889, "chain_loss": null, "readout_loss": 1.4542791602949872, "readout_argmax_matches_target": 0.3114754098360656, "readout_mean_top_prob": 0.579077528881245, "readout_target_entropy": 0.024335333069816965, "mem_gb": 17.69098752}
{"step": 3, "of": 37, "lr": 9.924038765061042e-05, "grad_norm": 6.122931957244873, "examples": 123, "readout_examples": 123, "reasoner_examples": 0, "step_tokens": 44319, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 3321.107801756901, "elapsed_min": 0.6973576704661052, "eta_min": 7.903386931949192, "chain_loss": null, "readout_loss": 0.906463114720792, "readout_argmax_matches_target": 0.6504065040650406, "readout_mean_top_prob": 0.7174172257262517, "readout_target_entropy": 0.03210654685984089, "mem_gb": 17.69098752}
{"step": 10, "of": 37, "lr": 8.535533905932738e-05, "grad_norm": 9.12912654876709, "examples": 56, "readout_examples": 56, "reasoner_examples": 0, "step_tokens": 47335, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 2516.7677815736165, "elapsed_min": 3.0938889384269714, "eta_min": 8.353500133752823, "chain_loss": null, "readout_loss": 1.0244907473596478, "readout_argmax_matches_target": 0.5714285714285714, "readout_mean_top_prob": 0.6686786732503346, "readout_target_entropy": 0.022231058284108127, "mem_gb": 17.69098752}
{"step": 20, "of": 37, "lr": 4.564221286261709e-05, "grad_norm": 5.417482376098633, "examples": 97, "readout_examples": 97, "reasoner_examples": 0, "step_tokens": 46614, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 2148.288201561984, "elapsed_min": 7.265055648485819, "eta_min": 6.1752973012129475, "chain_loss": null, "readout_loss": 1.0702674461038995, "readout_argmax_matches_target": 0.5876288659793815, "readout_mean_top_prob": 0.6540040390700409, "readout_target_entropy": 0.010466781596067487, "mem_gb": 17.69098752}
{"step": 30, "of": 37, "lr": 9.042397785550405e-06, "grad_norm": 4.825333118438721, "examples": 68, "readout_examples": 68, "reasoner_examples": 0, "step_tokens": 47388, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 2341.9283423613306, "elapsed_min": 9.970871543884277, "eta_min": 2.326536693572998, "chain_loss": null, "readout_loss": 0.5516520653139142, "readout_argmax_matches_target": 0.8970588235294118, "readout_mean_top_prob": 0.7004640414653456, "readout_target_entropy": 0.14180531255517373, "mem_gb": 17.69098752}
{"mode": "readout_only", "alpha": 1.0, "lr": 0.0001, "rank": 32, "lora_alpha": 64, "seed": 17, "examples": 3169, "steps_done": 37, "steps_planned": 37, "stopped_at_deadline": false, "train_tokens": 1711105, "train_seconds": 678.7952787876129, "tok_per_s": 2520.7968491710512, "counts": {"readout_from_votes": 3162, "readout_gold_fallback": 7}, "reasoner_examples": 0, "readout_examples": 3169}
{"time": "2026-10-31T07:10:15Z", "event": "done", "stage": "train_gold", "hours_left": 2.33}
{"time": "2026-10-31T07:10:38Z", "event": "start", "stage": "merge_gold", "hours_left": 2.32}
Loading weights: 0%| | 0/426 [00:00<?, ?it/s]Loading weights: 84%|████████▎ | 356/426 [00:00<00:00, 3531.17it/s]Loading weights: 100%|██████████| 426/426 [00:00<00:00, 3534.91it/s]
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]Writing model shards: 100%|██████████| 1/1 [00:03<00:00, 3.62s/it]Writing model shards: 100%|██████████| 1/1 [00:03<00:00, 3.62s/it]
merged results/adapters/gold_only into checkpoints/gold_only_merged
{"time": "2026-10-31T07:11:40Z", "event": "done", "stage": "merge_gold", "hours_left": 2.3}
{"time": "2026-10-31T07:12:00Z", "event": "start", "stage": "train_votes", "hours_left": 2.3}
{"examples": 3169, "tokens": 1711105, "readout_from_votes": 3162, "readout_gold_fallback": 7}
Loading weights: 0%| | 0/426 [00:00<?, ?it/s]Loading weights: 88%|████████▊ | 374/426 [00:00<00:00, 3736.46it/s]Loading weights: 100%|██████████| 426/426 [00:00<00:00, 3690.31it/s]
trainable params: 60,948,480 || all params: 4,266,699,776 || trainable%: 1.4285
[transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
/workspace/decision-reasoner/train.py:145: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
Consider using tensor.detach() first. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:822.)
stats["readout_loss"] += float(loss)
{"probe_tok_per_s": 3838.0248403195706, "fit_tokens": 5899450.747378086, "have_tokens": 1711105}
{"step": 1, "of": 37, "lr": 0.0001, "grad_norm": 16.499691009521484, "examples": 50, "readout_examples": 50, "reasoner_examples": 0, "step_tokens": 47327, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 3962.3054761679205, "elapsed_min": 0.1990718126296997, "eta_min": 7.166585254669189, "chain_loss": null, "readout_loss": 1.439049574509263, "readout_argmax_matches_target": 0.44, "readout_mean_top_prob": 0.5607822763919831, "readout_target_entropy": 0.11168528795242309, "mem_gb": 17.69098752}
{"step": 2, "of": 37, "lr": 9.980973490458728e-05, "grad_norm": 20.690614700317383, "examples": 61, "readout_examples": 61, "reasoner_examples": 0, "step_tokens": 47314, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 4012.912072142604, "elapsed_min": 0.3930686672528585, "eta_min": 6.878701676925023, "chain_loss": null, "readout_loss": 1.4830484572401057, "readout_argmax_matches_target": 0.32786885245901637, "readout_mean_top_prob": 0.579077528881245, "readout_target_entropy": 0.0866156288834869, "mem_gb": 17.69098752}
{"step": 3, "of": 37, "lr": 9.924038765061042e-05, "grad_norm": 5.102482795715332, "examples": 123, "readout_examples": 123, "reasoner_examples": 0, "step_tokens": 44319, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 3912.300240803209, "elapsed_min": 0.5919791062672933, "eta_min": 6.70909653769599, "chain_loss": null, "readout_loss": 0.720007419059, "readout_argmax_matches_target": 0.7073170731707317, "readout_mean_top_prob": 0.7052061976940651, "readout_target_entropy": 0.12027148987219585, "mem_gb": 17.69098752}
{"step": 10, "of": 37, "lr": 8.535533905932738e-05, "grad_norm": 8.605474472045898, "examples": 56, "readout_examples": 56, "reasoner_examples": 0, "step_tokens": 47335, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 3920.875668113465, "elapsed_min": 1.9859339237213134, "eta_min": 5.362021594047547, "chain_loss": null, "readout_loss": 0.9844543824458795, "readout_argmax_matches_target": 0.6071428571428571, "readout_mean_top_prob": 0.6930059604346752, "readout_target_entropy": 0.08096023542540413, "mem_gb": 17.69098752}
{"step": 20, "of": 37, "lr": 4.564221286261709e-05, "grad_norm": 6.041411399841309, "examples": 97, "readout_examples": 97, "reasoner_examples": 0, "step_tokens": 46614, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 3916.0817292546385, "elapsed_min": 3.98547180891037, "eta_min": 3.387651037573814, "chain_loss": null, "readout_loss": 0.930043653974468, "readout_argmax_matches_target": 0.6494845360824743, "readout_mean_top_prob": 0.7174164949311423, "readout_target_entropy": 0.1419014734091218, "mem_gb": 17.69098752}
{"step": 30, "of": 37, "lr": 9.042397785550405e-06, "grad_norm": 4.9224772453308105, "examples": 68, "readout_examples": 68, "reasoner_examples": 0, "step_tokens": 47388, "chain_tokens": 0, "micro_batches": 3, "tok_per_s": 3910.8449548034732, "elapsed_min": 5.970849506060282, "eta_min": 1.3931982180807325, "chain_loss": null, "readout_loss": 0.43335475402829404, "readout_argmax_matches_target": 0.8529411764705882, "readout_mean_top_prob": 0.7824242110638058, "readout_target_entropy": 0.043113489361370314, "mem_gb": 17.69098752}
{"mode": "readout_only", "alpha": 0.0, "lr": 0.0001, "rank": 32, "lora_alpha": 64, "seed": 17, "examples": 3169, "steps_done": 37, "steps_planned": 37, "stopped_at_deadline": false, "train_tokens": 1711105, "train_seconds": 438.92514300346375, "tok_per_s": 3898.3982286621867, "counts": {"readout_from_votes": 3162, "readout_gold_fallback": 7}, "reasoner_examples": 0, "readout_examples": 3169}
{"time": "2026-10-31T07:21:52Z", "event": "done", "stage": "train_votes", "hours_left": 2.13}
{"time": "2026-10-31T07:22:20Z", "event": "start", "stage": "merge_votes", "hours_left": 2.12}
Loading weights: 0%| | 0/426 [00:00<?, ?it/s]Loading weights: 86%|████████▌ | 366/426 [00:00<00:00, 3646.13it/s]Loading weights: 100%|██████████| 426/426 [00:00<00:00, 3602.66it/s]
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]Writing model shards: 100%|██████████| 1/1 [00:08<00:00, 8.13s/it]Writing model shards: 100%|██████████| 1/1 [00:08<00:00, 8.13s/it]
merged results/adapters/votes_only into checkpoints/votes_only_merged
{"time": "2026-10-31T07:23:24Z", "event": "done", "stage": "merge_votes", "hours_left": 2.11}
{"time": "2026-10-31T07:23:44Z", "event": "start", "stage": "eval_joint", "hours_left": 2.1}
(MainProcess pid=50120) INFO 10-31 07:23:55 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM
(MainProcess pid=50120) INFO 10-31 07:23:55 [model.py:2152] Using max model len 32768
(MainProcess pid=50120) INFO 10-31 07:23:55 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=50120) INFO 10-31 07:23:55 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled
(MainProcess pid=50120) INFO 10-31 07:23:55 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=50398) INFO 10-31 07:24:01 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/joint_merged', speculative_config=None, tokenizer='checkpoints/joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=50398) INFO 10-31 07:24:02 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_2dee883a0b7446c7828a1f57565dc0d8 backend=nccl
(EngineCore pid=50398) INFO 10-31 07:24:02 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=50398) INFO 10-31 07:24:02 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=50398) INFO 10-31 07:24:04 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=50398) INFO 10-31 07:24:04 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=50398) INFO 10-31 07:24:05 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=50398) INFO 10-31 07:24:05 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=50398) INFO 10-31 07:24:05 [base_loader.py:82] Initializing model took 1.592674 seconds
(EngineCore pid=50398) INFO 10-31 07:24:05 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 169.09 GiB.
(EngineCore pid=50398) INFO 10-31 07:24:05 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.23it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.23it/s]
(EngineCore pid=50398) INFO 10-31 07:24:06 [default_loader.py:501] Loading weights took 0.84 seconds
(EngineCore pid=50398) INFO 10-31 07:24:08 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.422364 seconds
(EngineCore pid=50398) INFO 10-31 07:24:09 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=50398) INFO 10-31 07:24:09 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=50398) INFO 10-31 07:24:09 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=50398) INFO 10-31 07:24:09 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=50398) INFO 10-31 07:24:09 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=50398) INFO 10-31 07:24:10 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=50398) INFO 10-31 07:24:15 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/c228b0ef18/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=50398) INFO 10-31 07:24:15 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.45 s
(EngineCore pid=50398) INFO 10-31 07:24:31 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 15.85 s
(EngineCore pid=50398) INFO 10-31 07:24:34 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20434244 bytes total
(EngineCore pid=50398) INFO 10-31 07:24:34 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/a0756fe669bc0fbaca33b91d233bcfa2f00d9afbd58c548177a6ef2f7d1ad0b0/rank_0_0/model
(EngineCore pid=50398) INFO 10-31 07:24:34 [monitor.py:62] [backbone] torch.compile took 24.27 s in total
(EngineCore pid=50398) INFO 10-31 07:26:00 [monitor.py:91] [backbone] Initial profiling/warmup run took 86.39 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:01<00:51, 1.03s/it]Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:04<01:51, 2.28s/it]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:04<01:09, 1.45s/it]Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:04<00:47, 1.01s/it]Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:05<00:35, 1.28it/s]Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:05<00:29, 1.54it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:06<00:27, 1.63it/s]Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:06<00:24, 1.73it/s]Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:09<00:57, 1.36s/it]Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:10<00:42, 1.03s/it]Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:10<00:32, 1.25it/s]Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:10<00:25, 1.50it/s]Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:11<00:21, 1.79it/s]Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:11<00:17, 2.08it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:11<00:14, 2.51it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:11<00:12, 2.91it/s]Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:12<00:18, 1.80it/s]Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:13<00:15, 2.14it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:13<00:11, 2.74it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:13<00:05, 5.35it/s]Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:13<00:03, 8.01it/s]Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:13<00:02, 8.74it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:13<00:02, 10.46it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:13<00:01, 12.99it/s]Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:14<00:01, 14.12it/s]Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:14<00:01, 12.48it/s]Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:14<00:00, 13.95it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:14<00:00, 15.84it/s]Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:14<00:00, 13.91it/s]Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:14<00:00, 14.89it/s]Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:14<00:00, 16.73it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:15<00:00, 13.48it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:15<00:00, 3.33it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 26.42it/s]
(EngineCore pid=50398) INFO 10-31 07:26:19 [model_runner.py:1127] Graph capturing finished in 16 secs, took 0.40 GiB
(EngineCore pid=50398) INFO 10-31 07:26:19 [gpu_worker.py:759] Available KV cache memory: 59.1 GiB
(EngineCore pid=50398) INFO 10-31 07:26:19 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8940 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9060. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=50398) INFO 10-31 07:26:19 [kv_cache_utils.py:2327] GPU KV cache size: 1,741,452 tokens, Maximum concurrency for 32,768 tokens per request: 53.14x
(EngineCore pid=50398) INFO 10-31 07:26:19 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=50398) INFO 10-31 07:26:19 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=50398) INFO 10-31 07:26:19 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=50398) INFO 10-31 07:26:19 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=50398) INFO 10-31 07:26:19 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=50398) INFO 10-31 07:26:21 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:41, 1.22it/s]Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:40, 1.22it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:38, 1.24it/s]Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:36, 1.30it/s]Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:03<00:33, 1.38it/s]Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:04<00:30, 1.46it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:04<00:28, 1.56it/s]Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:25, 1.68it/s]Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:05<00:22, 1.83it/s]Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:20, 2.00it/s]Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:18, 2.21it/s]Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:15, 2.49it/s]Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:15, 2.48it/s]Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:13, 2.78it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:12, 2.93it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:08<00:09, 3.54it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:08<00:04, 6.69it/s]Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:08<00:04, 6.91it/s]Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:08<00:02, 10.29it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:08<00:01, 13.07it/s]Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:01, 13.16it/s]Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:08<00:01, 12.88it/s]Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:09<00:01, 15.13it/s]Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:09<00:00, 16.70it/s]Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:09<00:00, 15.02it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:09<00:00, 16.86it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:09<00:00, 18.18it/s]Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:09<00:00, 15.01it/s]Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:09<00:00, 16.88it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 5.08it/s]
Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 28.10it/s]Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 21.27it/s]Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 23.77it/s]Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:00, 25.13it/s]Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 25.78it/s]Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 22.98it/s]Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 24.42it/s]Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:00<00:00, 25.30it/s]Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 26.23it/s]Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 22.60it/s]Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 23.28it/s]Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 24.13it/s]
(EngineCore pid=50398) INFO 10-31 07:26:52 [model_runner.py:1127] Graph capturing finished in 13 secs, took 0.24 GiB
(EngineCore pid=50398) INFO 10-31 07:26:52 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (99.2%).
(EngineCore pid=50398) INFO 10-31 07:26:52 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 10.52 GiB for consumed memory (weights + non-torch), 1.71 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=63044161434` (58.71 GiB) to fit into requested memory, or `--kv-cache-memory=71110227456` (66.23 GiB) to fully utilize gpu memory. Current kv cache memory in use is 59.1 GiB.
(EngineCore pid=50398) INFO 10-31 07:26:52 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=50398) INFO 10-31 07:26:53 [core.py:395] init engine (profile, create kv cache, warmup model) took 164.06 s (compilation: 24.27 s)
(EngineCore pid=50398) INFO 10-31 07:26:53 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=50120) WARNING 10-31 07:26:54 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=50398) WARNING 10-31 07:26:54 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout joint: 5957 items in 113s
(EngineCore pid=50398) WARNING 10-31 07:28:47 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=50398) WARNING 10-31 07:28:47 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=50398) WARNING 10-31 07:28:49 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
[rank0]:[W1031 07:29:03.201210575 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 245366784 bytes (free: 205389824, total: 85094825984).
progress {"time": "07:29:46", "elapsed_min": 1.0, "submitted": 323, "finished": 67, "in_flight": 256, "gen_tokens": 46544, "tok_per_s": 776, "chain_tokens_median": 708.0, "chain_tokens_p90": 869.4, "chain_tokens_max": 951, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:29:46 [loggers.py:335] Engine 000: Avg prompt throughput: 17008.8 tokens/s, Avg generation throughput: 1397.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.1%, Prefix cache hit rate: 2.0%
progress {"time": "07:30:46", "elapsed_min": 2.0, "submitted": 411, "finished": 155, "in_flight": 256, "gen_tokens": 152883, "tok_per_s": 1274, "chain_tokens_median": 870.0, "chain_tokens_p90": 1637.3999999999999, "chain_tokens_max": 1944, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:30:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3354.1 tokens/s, Avg generation throughput: 4326.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.1%, Prefix cache hit rate: 5.1%
progress {"time": "07:31:46", "elapsed_min": 3.0, "submitted": 489, "finished": 233, "in_flight": 256, "gen_tokens": 279612, "tok_per_s": 1553, "chain_tokens_median": 954.0, "chain_tokens_p90": 2206.2000000000007, "chain_tokens_max": 2819, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:31:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3902.0 tokens/s, Avg generation throughput: 3763.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 58.8%, Prefix cache hit rate: 6.7%
progress {"time": "07:32:46", "elapsed_min": 4.0, "submitted": 589, "finished": 333, "in_flight": 256, "gen_tokens": 417903, "tok_per_s": 1741, "chain_tokens_median": 914.0, "chain_tokens_p90": 2622.2000000000003, "chain_tokens_max": 3626, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:32:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3775.4 tokens/s, Avg generation throughput: 3535.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 61.6%, Prefix cache hit rate: 7.6%
progress {"time": "07:33:46", "elapsed_min": 5.0, "submitted": 679, "finished": 423, "in_flight": 256, "gen_tokens": 570512, "tok_per_s": 1902, "chain_tokens_median": 911.0, "chain_tokens_p90": 2939.0, "chain_tokens_max": 4448, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:33:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3862.0 tokens/s, Avg generation throughput: 3374.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 63.8%, Prefix cache hit rate: 7.0%
progress {"time": "07:34:46", "elapsed_min": 6.0, "submitted": 758, "finished": 502, "in_flight": 256, "gen_tokens": 744444, "tok_per_s": 2068, "chain_tokens_median": 931.5, "chain_tokens_p90": 3435.2000000000007, "chain_tokens_max": 5176, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:34:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3989.0 tokens/s, Avg generation throughput: 3302.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.3%, Prefix cache hit rate: 7.2%
progress {"time": "07:35:46", "elapsed_min": 7.0, "submitted": 833, "finished": 577, "in_flight": 256, "gen_tokens": 872988, "tok_per_s": 2078, "chain_tokens_median": 930.0, "chain_tokens_p90": 3548.2, "chain_tokens_max": 5974, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:35:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2934.6 tokens/s, Avg generation throughput: 3631.9 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.8%, Prefix cache hit rate: 6.1%
progress {"time": "07:36:46", "elapsed_min": 8.0, "submitted": 911, "finished": 655, "in_flight": 256, "gen_tokens": 1041235, "tok_per_s": 2169, "chain_tokens_median": 951.0, "chain_tokens_p90": 3695.8, "chain_tokens_max": 6791, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:36:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3802.0 tokens/s, Avg generation throughput: 3208.0 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.6%, Prefix cache hit rate: 4.3%
progress {"time": "07:37:46", "elapsed_min": 9.0, "submitted": 987, "finished": 731, "in_flight": 256, "gen_tokens": 1236325, "tok_per_s": 2289, "chain_tokens_median": 974.0, "chain_tokens_p90": 3969.0, "chain_tokens_max": 7508, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:37:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4333.7 tokens/s, Avg generation throughput: 2995.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.9%, Prefix cache hit rate: 3.0%
progress {"time": "07:38:46", "elapsed_min": 10.0, "submitted": 1060, "finished": 804, "in_flight": 256, "gen_tokens": 1432867, "tok_per_s": 2388, "chain_tokens_median": 1014.0, "chain_tokens_p90": 4231.600000000002, "chain_tokens_max": 7982, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:38:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4310.3 tokens/s, Avg generation throughput: 3084.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.9%, Prefix cache hit rate: 1.8%
progress {"time": "07:39:46", "elapsed_min": 11.0, "submitted": 1132, "finished": 876, "in_flight": 256, "gen_tokens": 1601814, "tok_per_s": 2427, "chain_tokens_median": 1005.5, "chain_tokens_p90": 4388.5, "chain_tokens_max": 8959, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:39:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3864.1 tokens/s, Avg generation throughput: 3055.0 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.3%, Prefix cache hit rate: 1.2%
progress {"time": "07:40:46", "elapsed_min": 12.0, "submitted": 1212, "finished": 956, "in_flight": 256, "gen_tokens": 1782507, "tok_per_s": 2476, "chain_tokens_median": 996.0, "chain_tokens_p90": 4443.0, "chain_tokens_max": 9298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:40:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4023.8 tokens/s, Avg generation throughput: 3053.9 tokens/s, Running: 253 reqs, Waiting: 2 reqs, GPU KV cache usage: 67.0%, Prefix cache hit rate: 1.1%
progress {"time": "07:41:46", "elapsed_min": 13.0, "submitted": 1313, "finished": 1057, "in_flight": 256, "gen_tokens": 1951521, "tok_per_s": 2502, "chain_tokens_median": 949.0, "chain_tokens_p90": 4460.6, "chain_tokens_max": 9845, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:41:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3915.1 tokens/s, Avg generation throughput: 2993.0 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.6%, Prefix cache hit rate: 0.5%
progress {"time": "07:42:46", "elapsed_min": 14.0, "submitted": 1418, "finished": 1162, "in_flight": 256, "gen_tokens": 2124699, "tok_per_s": 2529, "chain_tokens_median": 912.5, "chain_tokens_p90": 4455.200000000001, "chain_tokens_max": 10721, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:42:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4457.4 tokens/s, Avg generation throughput: 2746.6 tokens/s, Running: 255 reqs, Waiting: 1 reqs, GPU KV cache usage: 67.6%, Prefix cache hit rate: 0.4%
progress {"time": "07:43:46", "elapsed_min": 15.0, "submitted": 1496, "finished": 1240, "in_flight": 256, "gen_tokens": 2244199, "tok_per_s": 2493, "chain_tokens_median": 901.0, "chain_tokens_p90": 4448.800000000001, "chain_tokens_max": 10721, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:43:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3324.6 tokens/s, Avg generation throughput: 3208.5 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.6%, Prefix cache hit rate: 0.6%
progress {"time": "07:44:46", "elapsed_min": 16.0, "submitted": 1569, "finished": 1313, "in_flight": 256, "gen_tokens": 2405334, "tok_per_s": 2505, "chain_tokens_median": 918.0, "chain_tokens_p90": 4492.5999999999985, "chain_tokens_max": 11957, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:44:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3674.2 tokens/s, Avg generation throughput: 3166.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.6%, Prefix cache hit rate: 0.8%
progress {"time": "07:45:46", "elapsed_min": 17.0, "submitted": 1670, "finished": 1414, "in_flight": 256, "gen_tokens": 2593630, "tok_per_s": 2543, "chain_tokens_median": 917.5, "chain_tokens_p90": 4573.4, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:45:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4271.7 tokens/s, Avg generation throughput: 2753.6 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.8%
progress {"time": "07:46:46", "elapsed_min": 18.0, "submitted": 1760, "finished": 1504, "in_flight": 256, "gen_tokens": 2757289, "tok_per_s": 2553, "chain_tokens_median": 915.0, "chain_tokens_p90": 4573.4, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:46:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3646.6 tokens/s, Avg generation throughput: 3079.8 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.8%, Prefix cache hit rate: 0.8%
progress {"time": "07:47:46", "elapsed_min": 19.0, "submitted": 1828, "finished": 1572, "in_flight": 256, "gen_tokens": 2914911, "tok_per_s": 2557, "chain_tokens_median": 936.5, "chain_tokens_p90": 4575.8, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:47:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3826.4 tokens/s, Avg generation throughput: 2974.2 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.9%
progress {"time": "07:48:46", "elapsed_min": 20.0, "submitted": 1895, "finished": 1639, "in_flight": 256, "gen_tokens": 3053029, "tok_per_s": 2544, "chain_tokens_median": 939.0, "chain_tokens_p90": 4585.800000000001, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:48:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3096.4 tokens/s, Avg generation throughput: 3205.8 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.9%, Prefix cache hit rate: 1.2%
progress {"time": "07:49:46", "elapsed_min": 21.0, "submitted": 1973, "finished": 1717, "in_flight": 256, "gen_tokens": 3159228, "tok_per_s": 2507, "chain_tokens_median": 923.0, "chain_tokens_p90": 4572.8, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:49:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2989.7 tokens/s, Avg generation throughput: 2867.7 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.4%, Prefix cache hit rate: 1.3%
progress {"time": "07:50:46", "elapsed_min": 22.0, "submitted": 2047, "finished": 1791, "in_flight": 256, "gen_tokens": 3311368, "tok_per_s": 2509, "chain_tokens_median": 917.0, "chain_tokens_p90": 4576.0, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:50:46 [loggers.py:335] Engine 000: Avg prompt throughput: 4242.5 tokens/s, Avg generation throughput: 2534.0 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.8%, Prefix cache hit rate: 0.8%
progress {"time": "07:51:46", "elapsed_min": 23.0, "submitted": 2098, "finished": 1842, "in_flight": 256, "gen_tokens": 3480011, "tok_per_s": 2522, "chain_tokens_median": 941.5, "chain_tokens_p90": 4612.300000000001, "chain_tokens_max": 13116, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:51:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3543.7 tokens/s, Avg generation throughput: 2958.7 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.2%, Prefix cache hit rate: 0.6%
progress {"time": "07:52:46", "elapsed_min": 24.0, "submitted": 2160, "finished": 1904, "in_flight": 256, "gen_tokens": 3607304, "tok_per_s": 2505, "chain_tokens_median": 939.5, "chain_tokens_p90": 4608.900000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:52:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2999.3 tokens/s, Avg generation throughput: 2779.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.3%, Prefix cache hit rate: 0.7%
progress {"time": "07:53:46", "elapsed_min": 25.0, "submitted": 2223, "finished": 1967, "in_flight": 256, "gen_tokens": 3742421, "tok_per_s": 2495, "chain_tokens_median": 941.0, "chain_tokens_p90": 4620.400000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:53:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3278.4 tokens/s, Avg generation throughput: 2804.8 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.1%, Prefix cache hit rate: 0.6%
progress {"time": "07:54:46", "elapsed_min": 26.0, "submitted": 2289, "finished": 2033, "in_flight": 256, "gen_tokens": 3895368, "tok_per_s": 2497, "chain_tokens_median": 952.0, "chain_tokens_p90": 4633.0, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:54:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3650.6 tokens/s, Avg generation throughput: 2552.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.8%, Prefix cache hit rate: 0.6%
progress {"time": "07:55:46", "elapsed_min": 27.0, "submitted": 2365, "finished": 2109, "in_flight": 256, "gen_tokens": 4016987, "tok_per_s": 2480, "chain_tokens_median": 942.0, "chain_tokens_p90": 4625.8, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:55:46 [loggers.py:335] Engine 000: Avg prompt throughput: 3179.1 tokens/s, Avg generation throughput: 2658.5 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 0.2%
progress {"time": "07:56:46", "elapsed_min": 28.0, "submitted": 2435, "finished": 2179, "in_flight": 256, "gen_tokens": 4131815, "tok_per_s": 2459, "chain_tokens_median": 941.0, "chain_tokens_p90": 4619.200000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:56:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2816.6 tokens/s, Avg generation throughput: 2821.5 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.9%, Prefix cache hit rate: 0.2%
progress {"time": "07:57:46", "elapsed_min": 29.0, "submitted": 2512, "finished": 2256, "in_flight": 256, "gen_tokens": 4240220, "tok_per_s": 2437, "chain_tokens_median": 936.5, "chain_tokens_p90": 4579.5, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:57:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2949.3 tokens/s, Avg generation throughput: 2650.5 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.0%, Prefix cache hit rate: 0.0%
progress {"time": "07:58:46", "elapsed_min": 30.0, "submitted": 2569, "finished": 2313, "in_flight": 256, "gen_tokens": 4341840, "tok_per_s": 2412, "chain_tokens_median": 939.0, "chain_tokens_p90": 4617.200000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:58:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2860.4 tokens/s, Avg generation throughput: 2557.7 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.0%, Prefix cache hit rate: 0.0%
progress {"time": "07:59:46", "elapsed_min": 31.0, "submitted": 2611, "finished": 2355, "in_flight": 256, "gen_tokens": 4415281, "tok_per_s": 2374, "chain_tokens_median": 941.0, "chain_tokens_p90": 4591.399999999999, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 07:59:46 [loggers.py:335] Engine 000: Avg prompt throughput: 1945.8 tokens/s, Avg generation throughput: 2746.8 tokens/s, Running: 250 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 0.0%
progress {"time": "08:00:46", "elapsed_min": 32.0, "submitted": 2616, "finished": 2360, "in_flight": 256, "gen_tokens": 4436745, "tok_per_s": 2311, "chain_tokens_median": 941.0, "chain_tokens_p90": 4618.599999999999, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:00:46 [loggers.py:335] Engine 000: Avg prompt throughput: 442.3 tokens/s, Avg generation throughput: 2340.3 tokens/s, Running: 220 reqs, Waiting: 35 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0%
progress {"time": "08:01:46", "elapsed_min": 33.0, "submitted": 2656, "finished": 2400, "in_flight": 256, "gen_tokens": 4517481, "tok_per_s": 2281, "chain_tokens_median": 950.5, "chain_tokens_p90": 4618.599999999999, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:01:46 [loggers.py:335] Engine 000: Avg prompt throughput: 1925.8 tokens/s, Avg generation throughput: 2034.0 tokens/s, Running: 205 reqs, Waiting: 51 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 0.0%
progress {"time": "08:02:46", "elapsed_min": 34.0, "submitted": 2683, "finished": 2427, "in_flight": 256, "gen_tokens": 4610168, "tok_per_s": 2260, "chain_tokens_median": 956.0, "chain_tokens_p90": 4644.6, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:02:46 [loggers.py:335] Engine 000: Avg prompt throughput: 1923.7 tokens/s, Avg generation throughput: 1885.7 tokens/s, Running: 200 reqs, Waiting: 54 reqs, GPU KV cache usage: 98.0%, Prefix cache hit rate: 0.0%
progress {"time": "08:03:46", "elapsed_min": 35.0, "submitted": 2713, "finished": 2457, "in_flight": 256, "gen_tokens": 4704725, "tok_per_s": 2240, "chain_tokens_median": 970.0, "chain_tokens_p90": 4684.800000000001, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:03:46 [loggers.py:335] Engine 000: Avg prompt throughput: 2226.1 tokens/s, Avg generation throughput: 1887.7 tokens/s, Running: 200 reqs, Waiting: 56 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0%
progress {"time": "08:04:46", "elapsed_min": 36.0, "submitted": 2715, "finished": 2459, "in_flight": 256, "gen_tokens": 4717543, "tok_per_s": 2184, "chain_tokens_median": 973.0, "chain_tokens_p90": 4728.600000000005, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:04:46 [loggers.py:335] Engine 000: Avg prompt throughput: 350.0 tokens/s, Avg generation throughput: 2303.6 tokens/s, Running: 183 reqs, Waiting: 73 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0%
progress {"time": "08:05:46", "elapsed_min": 37.0, "submitted": 2717, "finished": 2461, "in_flight": 256, "gen_tokens": 4720605, "tok_per_s": 2126, "chain_tokens_median": 973.0, "chain_tokens_p90": 4725.0, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:05:46 [loggers.py:335] Engine 000: Avg prompt throughput: 87.2 tokens/s, Avg generation throughput: 2105.9 tokens/s, Running: 170 reqs, Waiting: 86 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 0.0%
progress {"time": "08:06:46", "elapsed_min": 38.0, "submitted": 2736, "finished": 2480, "in_flight": 256, "gen_tokens": 4779993, "tok_per_s": 2096, "chain_tokens_median": 976.5, "chain_tokens_p90": 4755.5, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:06:46 [loggers.py:335] Engine 000: Avg prompt throughput: 1208.2 tokens/s, Avg generation throughput: 1816.0 tokens/s, Running: 153 reqs, Waiting: 103 reqs, GPU KV cache usage: 99.3%, Prefix cache hit rate: 0.0%
progress {"time": "08:07:46", "elapsed_min": 39.0, "submitted": 2819, "finished": 2563, "in_flight": 256, "gen_tokens": 5068653, "tok_per_s": 2166, "chain_tokens_median": 1004.0, "chain_tokens_p90": 4854.800000000003, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:07:46 [loggers.py:335] Engine 000: Avg prompt throughput: 5749.3 tokens/s, Avg generation throughput: 1155.6 tokens/s, Running: 171 reqs, Waiting: 85 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0%
progress {"time": "08:08:46", "elapsed_min": 40.0, "submitted": 2819, "finished": 2563, "in_flight": 256, "gen_tokens": 5068653, "tok_per_s": 2112, "chain_tokens_median": 1004.0, "chain_tokens_p90": 4854.800000000003, "chain_tokens_max": 17978, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=50120) INFO 10-31 08:08:46 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1909.1 tokens/s, Running: 140 reqs, Waiting: 116 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0%
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "prep", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "probe", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "baseline", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "generate", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "train_joint", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "merge_joint", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "train_gold", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "merge_gold", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "train_votes", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "skip", "stage": "merge_votes", "hours_left": 1.33}
{"time": "2026-10-31T08:09:52Z", "event": "start", "stage": "eval_joint", "hours_left": 1.33}
(MainProcess pid=52652) INFO 10-31 08:10:04 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM
(MainProcess pid=52652) INFO 10-31 08:10:04 [model.py:2152] Using max model len 32768
(MainProcess pid=52652) INFO 10-31 08:10:04 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=52652) INFO 10-31 08:10:04 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled
(MainProcess pid=52652) INFO 10-31 08:10:04 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=52921) INFO 10-31 08:10:10 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/joint_merged', speculative_config=None, tokenizer='checkpoints/joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=52921) INFO 10-31 08:10:11 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_26d27b6bcf31426282aa10a426927f05 backend=nccl
(EngineCore pid=52921) INFO 10-31 08:10:11 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=52921) INFO 10-31 08:10:11 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=52921) INFO 10-31 08:10:13 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=52921) INFO 10-31 08:10:13 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=52921) INFO 10-31 08:10:14 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=52921) INFO 10-31 08:10:14 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=52921) INFO 10-31 08:10:17 [base_loader.py:82] Initializing model took 4.067586 seconds
(EngineCore pid=52921) INFO 10-31 08:10:17 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 163.58 GiB.
(EngineCore pid=52921) INFO 10-31 08:10:17 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.05it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.05it/s]
(EngineCore pid=52921) INFO 10-31 08:10:18 [default_loader.py:501] Loading weights took 0.98 seconds
(EngineCore pid=52921) INFO 10-31 08:10:19 [model_runner.py:424] Model loading took 7.94 GiB memory and 6.792969 seconds
(EngineCore pid=52921) INFO 10-31 08:10:20 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=52921) INFO 10-31 08:10:20 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=52921) INFO 10-31 08:10:20 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=52921) INFO 10-31 08:10:20 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=52921) INFO 10-31 08:10:20 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=52921) INFO 10-31 08:10:21 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=52921) INFO 10-31 08:10:27 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/764a5d377e/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=52921) INFO 10-31 08:10:27 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.29 s
(EngineCore pid=52921) INFO 10-31 08:10:43 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 16.27 s
(EngineCore pid=52921) INFO 10-31 08:10:46 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 26071923 bytes total
(EngineCore pid=52921) INFO 10-31 08:10:46 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/104e4206a2f7547583b1741445864a147db2dd2fea66f017ae75eae0a52768df/rank_0_0/model
(EngineCore pid=52921) INFO 10-31 08:10:46 [monitor.py:62] [backbone] torch.compile took 24.57 s in total
(EngineCore pid=52921) INFO 10-31 08:12:13 [monitor.py:91] [backbone] Initial profiling/warmup run took 87.29 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:32, 1.06it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:01<00:25, 1.29it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:18, 1.72it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:02<00:13, 2.22it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:02<00:12, 2.42it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:10, 2.87it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:03<00:09, 2.93it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:03<00:08, 3.25it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:08, 3.03it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:04<00:08, 2.88it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:04<00:07, 3.20it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:04<00:07, 3.24it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:05, 3.72it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:05<00:05, 3.72it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:05<00:04, 4.40it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:05<00:03, 5.09it/s]Capturing CUDA graphs (PIECEWISE): 49%|████▊ | 17/35 [00:06<00:08, 2.17it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:06<00:06, 2.47it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:09<00:12, 1.23it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:09<00:06, 1.95it/s]Capturing CUDA graphs (PIECEWISE): 71%|███████▏ | 25/35 [00:09<00:03, 3.23it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:12<00:06, 1.40it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:12<00:03, 2.03it/s]Capturing CUDA graphs (PIECEWISE): 89%|████████▊ | 31/35 [00:12<00:01, 3.28it/s]Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 33/35 [00:12<00:00, 4.30it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:12<00:00, 4.91it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:12<00:00, 2.74it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 25.79it/s]
(EngineCore pid=52921) INFO 10-31 08:12:29 [model_runner.py:1127] Graph capturing finished in 14 secs, took 0.30 GiB
(EngineCore pid=52921) INFO 10-31 08:12:30 [gpu_worker.py:759] Available KV cache memory: 59.37 GiB
(EngineCore pid=52921) INFO 10-31 08:12:30 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=52921) INFO 10-31 08:12:30 [kv_cache_utils.py:2327] GPU KV cache size: 1,749,526 tokens, Maximum concurrency for 32,768 tokens per request: 53.39x
(EngineCore pid=52921) INFO 10-31 08:12:30 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=52921) INFO 10-31 08:12:30 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=52921) INFO 10-31 08:12:30 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=52921) INFO 10-31 08:12:30 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=52921) INFO 10-31 08:12:30 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=52921) INFO 10-31 08:12:31 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:15, 2.15it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:11, 2.77it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:12, 2.64it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.63it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:10, 2.88it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:10, 2.68it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:09, 3.10it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:09, 2.84it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:08, 3.05it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:08, 2.79it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:07, 3.11it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:04<00:07, 2.96it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:06, 3.45it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:06, 3.19it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:05, 3.85it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:04, 4.59it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:05<00:02, 5.89it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:05<00:01, 7.54it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:05<00:01, 9.38it/s]Capturing CUDA graphs (PIECEWISE): 71%|███████▏ | 25/35 [00:05<00:00, 12.52it/s]Capturing CUDA graphs (PIECEWISE): 77%|███████▋ | 27/35 [00:05<00:00, 10.92it/s]Capturing CUDA graphs (PIECEWISE): 83%|████████▎ | 29/35 [00:06<00:00, 11.58it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:06<00:00, 14.27it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:06<00:00, 16.33it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:06<00:00, 5.56it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 11%|█ | 2/19 [00:00<00:01, 13.02it/s]Capturing CUDA graphs (FULL): 26%|██▋ | 5/19 [00:00<00:00, 18.88it/s]Capturing CUDA graphs (FULL): 42%|████▏ | 8/19 [00:00<00:00, 22.31it/s]Capturing CUDA graphs (FULL): 58%|█████▊ | 11/19 [00:00<00:00, 24.28it/s]Capturing CUDA graphs (FULL): 74%|███████▎ | 14/19 [00:00<00:00, 25.43it/s]Capturing CUDA graphs (FULL): 89%|████████▉ | 17/19 [00:00<00:00, 21.59it/s]Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 21.78it/s]
(EngineCore pid=52921) INFO 10-31 08:12:58 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.16 GiB
(EngineCore pid=52921) INFO 10-31 08:12:58 [gpu_worker.py:971] CUDA graph pool memory: 0.16 GiB (actual), 0.32 GiB (estimated), difference: 0.16 GiB (105.0%).
(EngineCore pid=52921) INFO 10-31 08:12:58 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 10.39 GiB for consumed memory (weights + non-torch), 1.56 GiB for peak activation, and 0.16 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=63427942810` (59.07 GiB) to fit into requested memory, or `--kv-cache-memory=71494008832` (66.58 GiB) to fully utilize gpu memory. Current kv cache memory in use is 59.37 GiB.
(EngineCore pid=52921) INFO 10-31 08:12:58 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=52921) INFO 10-31 08:12:59 [core.py:395] init engine (profile, create kv cache, warmup model) took 158.78 s (compilation: 24.57 s)
(EngineCore pid=52921) INFO 10-31 08:12:59 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
readout joint: exists
(MainProcess pid=52652) WARNING 10-31 08:13:00 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=52921) WARNING 10-31 08:13:01 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=52921) WARNING 10-31 08:13:02 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=52921) WARNING 10-31 08:13:03 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "08:13:59", "elapsed_min": 1.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:13:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1678.8 tokens/s, Avg generation throughput: 4172.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.0%
progress {"time": "08:14:59", "elapsed_min": 2.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:14:59 [loggers.py:335] Engine 000: Avg prompt throughput: 248.1 tokens/s, Avg generation throughput: 4617.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 38.0%, Prefix cache hit rate: 0.0%
(EngineCore pid=52921) WARNING 10-31 08:15:44 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "08:15:59", "elapsed_min": 3.0, "submitted": 343, "finished": 87, "in_flight": 256, "gen_tokens": 165975, "tok_per_s": 922, "chain_tokens_median": 1378.0, "chain_tokens_p90": 4092.0, "chain_tokens_max": 4697, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:15:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3332.2 tokens/s, Avg generation throughput: 3318.0 tokens/s, Running: 121 reqs, Waiting: 132 reqs, GPU KV cache usage: 40.1%, Prefix cache hit rate: 4.4%
progress {"time": "08:16:59", "elapsed_min": 4.0, "submitted": 394, "finished": 138, "in_flight": 256, "gen_tokens": 263082, "tok_per_s": 1096, "chain_tokens_median": 1068.5, "chain_tokens_p90": 4408.0, "chain_tokens_max": 5727, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:16:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2247.1 tokens/s, Avg generation throughput: 3514.4 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 41.4%, Prefix cache hit rate: 4.3%
progress {"time": "08:17:59", "elapsed_min": 5.0, "submitted": 538, "finished": 282, "in_flight": 256, "gen_tokens": 468930, "tok_per_s": 1563, "chain_tokens_median": 805.0, "chain_tokens_p90": 4581.500000000001, "chain_tokens_max": 7348, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:17:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4654.2 tokens/s, Avg generation throughput: 2732.7 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 39.3%, Prefix cache hit rate: 3.1%
progress {"time": "08:18:59", "elapsed_min": 6.0, "submitted": 668, "finished": 412, "in_flight": 256, "gen_tokens": 688748, "tok_per_s": 1913, "chain_tokens_median": 706.0, "chain_tokens_p90": 5116.300000000001, "chain_tokens_max": 8800, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:18:59 [loggers.py:335] Engine 000: Avg prompt throughput: 5184.1 tokens/s, Avg generation throughput: 2707.7 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 38.5%, Prefix cache hit rate: 2.5%
progress {"time": "08:19:59", "elapsed_min": 7.0, "submitted": 786, "finished": 530, "in_flight": 256, "gen_tokens": 894546, "tok_per_s": 2130, "chain_tokens_median": 727.5, "chain_tokens_p90": 5277.500000000001, "chain_tokens_max": 10487, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:19:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4759.2 tokens/s, Avg generation throughput: 2671.5 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 41.0%, Prefix cache hit rate: 2.4%
progress {"time": "08:20:59", "elapsed_min": 8.0, "submitted": 804, "finished": 548, "in_flight": 256, "gen_tokens": 927713, "tok_per_s": 1933, "chain_tokens_median": 738.5, "chain_tokens_p90": 5166.500000000002, "chain_tokens_max": 10849, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:20:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1168.9 tokens/s, Avg generation throughput: 3686.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 40.8%, Prefix cache hit rate: 1.6%
progress {"time": "08:21:59", "elapsed_min": 9.0, "submitted": 913, "finished": 657, "in_flight": 256, "gen_tokens": 1186905, "tok_per_s": 2198, "chain_tokens_median": 827.0, "chain_tokens_p90": 4835.999999999998, "chain_tokens_max": 12706, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:21:59 [loggers.py:335] Engine 000: Avg prompt throughput: 5622.2 tokens/s, Avg generation throughput: 2262.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 43.1%, Prefix cache hit rate: 1.2%
progress {"time": "08:22:59", "elapsed_min": 10.0, "submitted": 921, "finished": 665, "in_flight": 256, "gen_tokens": 1197698, "tok_per_s": 1996, "chain_tokens_median": 834.0, "chain_tokens_p90": 4762.400000000002, "chain_tokens_max": 12706, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:22:59 [loggers.py:335] Engine 000: Avg prompt throughput: 477.5 tokens/s, Avg generation throughput: 3661.7 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 48.1%, Prefix cache hit rate: 1.1%
progress {"time": "08:23:59", "elapsed_min": 11.0, "submitted": 932, "finished": 676, "in_flight": 256, "gen_tokens": 1223701, "tok_per_s": 1854, "chain_tokens_median": 849.0, "chain_tokens_p90": 4796.5, "chain_tokens_max": 12706, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:23:59 [loggers.py:335] Engine 000: Avg prompt throughput: 717.7 tokens/s, Avg generation throughput: 3409.8 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 50.6%, Prefix cache hit rate: 0.9%
progress {"time": "08:24:59", "elapsed_min": 12.0, "submitted": 961, "finished": 705, "in_flight": 256, "gen_tokens": 1306087, "tok_per_s": 1814, "chain_tokens_median": 891.0, "chain_tokens_p90": 4751.000000000002, "chain_tokens_max": 14338, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:24:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1795.3 tokens/s, Avg generation throughput: 2950.2 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.1%, Prefix cache hit rate: 0.9%
progress {"time": "08:25:59", "elapsed_min": 13.0, "submitted": 1042, "finished": 786, "in_flight": 256, "gen_tokens": 1652094, "tok_per_s": 2118, "chain_tokens_median": 989.0, "chain_tokens_p90": 5533.5, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:25:59 [loggers.py:335] Engine 000: Avg prompt throughput: 6602.1 tokens/s, Avg generation throughput: 1718.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.6%, Prefix cache hit rate: 0.7%
progress {"time": "08:26:59", "elapsed_min": 14.0, "submitted": 1051, "finished": 795, "in_flight": 256, "gen_tokens": 1699022, "tok_per_s": 2023, "chain_tokens_median": 1003.0, "chain_tokens_p90": 5605.600000000001, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:26:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1079.1 tokens/s, Avg generation throughput: 3151.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.8%, Prefix cache hit rate: 0.7%
progress {"time": "08:27:59", "elapsed_min": 15.0, "submitted": 1062, "finished": 806, "in_flight": 256, "gen_tokens": 1734071, "tok_per_s": 1927, "chain_tokens_median": 1026.5, "chain_tokens_p90": 5570.0, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:27:59 [loggers.py:335] Engine 000: Avg prompt throughput: 780.9 tokens/s, Avg generation throughput: 3200.8 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 55.8%, Prefix cache hit rate: 0.6%
progress {"time": "08:28:59", "elapsed_min": 16.0, "submitted": 1117, "finished": 861, "in_flight": 256, "gen_tokens": 1941549, "tok_per_s": 2022, "chain_tokens_median": 1098.0, "chain_tokens_p90": 5793.0, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:28:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3922.4 tokens/s, Avg generation throughput: 2171.8 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 57.4%, Prefix cache hit rate: 0.6%
progress {"time": "08:29:59", "elapsed_min": 17.0, "submitted": 1171, "finished": 915, "in_flight": 256, "gen_tokens": 2137071, "tok_per_s": 2095, "chain_tokens_median": 1141.0, "chain_tokens_p90": 5915.200000000001, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:29:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3689.8 tokens/s, Avg generation throughput: 2212.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.7%, Prefix cache hit rate: 0.5%
progress {"time": "08:30:59", "elapsed_min": 18.0, "submitted": 1180, "finished": 924, "in_flight": 256, "gen_tokens": 2171831, "tok_per_s": 2011, "chain_tokens_median": 1153.0, "chain_tokens_p90": 5916.900000000001, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:30:59 [loggers.py:335] Engine 000: Avg prompt throughput: 846.9 tokens/s, Avg generation throughput: 2884.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.5%, Prefix cache hit rate: 0.5%
progress {"time": "08:31:59", "elapsed_min": 19.0, "submitted": 1192, "finished": 936, "in_flight": 256, "gen_tokens": 2198587, "tok_per_s": 1929, "chain_tokens_median": 1156.0, "chain_tokens_p90": 5913.5, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:31:59 [loggers.py:335] Engine 000: Avg prompt throughput: 658.8 tokens/s, Avg generation throughput: 2936.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 62.6%, Prefix cache hit rate: 0.5%
progress {"time": "08:32:59", "elapsed_min": 20.0, "submitted": 1233, "finished": 977, "in_flight": 256, "gen_tokens": 2342154, "tok_per_s": 1952, "chain_tokens_median": 1192.0, "chain_tokens_p90": 5911.799999999999, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:32:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2665.8 tokens/s, Avg generation throughput: 2277.7 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 65.8%, Prefix cache hit rate: 0.4%
progress {"time": "08:33:59", "elapsed_min": 21.0, "submitted": 1300, "finished": 1044, "in_flight": 256, "gen_tokens": 2601133, "tok_per_s": 2064, "chain_tokens_median": 1291.5, "chain_tokens_p90": 6045.7, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:33:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4735.4 tokens/s, Avg generation throughput: 1738.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 67.6%, Prefix cache hit rate: 0.3%
progress {"time": "08:34:59", "elapsed_min": 22.0, "submitted": 1311, "finished": 1055, "in_flight": 256, "gen_tokens": 2638613, "tok_per_s": 1999, "chain_tokens_median": 1308.0, "chain_tokens_p90": 6045.6, "chain_tokens_max": 18055, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:34:59 [loggers.py:335] Engine 000: Avg prompt throughput: 838.6 tokens/s, Avg generation throughput: 2635.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 68.9%, Prefix cache hit rate: 0.2%
progress {"time": "08:35:59", "elapsed_min": 23.0, "submitted": 1325, "finished": 1069, "in_flight": 256, "gen_tokens": 2851339, "tok_per_s": 2066, "chain_tokens_median": 1340.0, "chain_tokens_p90": 6204.6, "chain_tokens_max": 31461, "parse_rate": 0.9943872778297475, "looped": 6, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:35:59 [loggers.py:335] Engine 000: Avg prompt throughput: 628.1 tokens/s, Avg generation throughput: 2738.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 62.5%, Prefix cache hit rate: 0.2%
progress {"time": "08:36:59", "elapsed_min": 24.0, "submitted": 1409, "finished": 1153, "in_flight": 256, "gen_tokens": 3526194, "tok_per_s": 2449, "chain_tokens_median": 1465.0, "chain_tokens_p90": 6371.799999999999, "chain_tokens_max": 32532, "parse_rate": 0.9826539462272333, "looped": 20, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:36:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4263.2 tokens/s, Avg generation throughput: 2219.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 40.9%, Prefix cache hit rate: 0.0%
progress {"time": "08:37:59", "elapsed_min": 25.0, "submitted": 1440, "finished": 1184, "in_flight": 256, "gen_tokens": 3626150, "tok_per_s": 2417, "chain_tokens_median": 1475.5, "chain_tokens_p90": 6357.5, "chain_tokens_max": 32532, "parse_rate": 0.9831081081081081, "looped": 20, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:37:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1989.0 tokens/s, Avg generation throughput: 2971.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 43.5%, Prefix cache hit rate: 0.0%
progress {"time": "08:38:59", "elapsed_min": 26.0, "submitted": 1453, "finished": 1197, "in_flight": 256, "gen_tokens": 3697685, "tok_per_s": 2370, "chain_tokens_median": 1490.0, "chain_tokens_p90": 6356.0, "chain_tokens_max": 32532, "parse_rate": 0.9824561403508771, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:38:59 [loggers.py:335] Engine 000: Avg prompt throughput: 932.8 tokens/s, Avg generation throughput: 3320.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 45.8%, Prefix cache hit rate: 0.0%
progress {"time": "08:39:59", "elapsed_min": 27.0, "submitted": 1526, "finished": 1270, "in_flight": 256, "gen_tokens": 3899922, "tok_per_s": 2407, "chain_tokens_median": 1523.5, "chain_tokens_p90": 6286.4000000000015, "chain_tokens_max": 32532, "parse_rate": 0.9834645669291339, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:39:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3934.1 tokens/s, Avg generation throughput: 2396.6 tokens/s, Running: 127 reqs, Waiting: 129 reqs, GPU KV cache usage: 47.8%, Prefix cache hit rate: 0.0%
progress {"time": "08:40:59", "elapsed_min": 28.0, "submitted": 1569, "finished": 1313, "in_flight": 256, "gen_tokens": 4028728, "tok_per_s": 2398, "chain_tokens_median": 1565.0, "chain_tokens_p90": 6236.599999999999, "chain_tokens_max": 32532, "parse_rate": 0.984006092916984, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:40:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2687.9 tokens/s, Avg generation throughput: 2658.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 49.6%, Prefix cache hit rate: 0.0%
progress {"time": "08:41:59", "elapsed_min": 29.0, "submitted": 1588, "finished": 1332, "in_flight": 256, "gen_tokens": 4082479, "tok_per_s": 2346, "chain_tokens_median": 1581.5, "chain_tokens_p90": 6213.700000000001, "chain_tokens_max": 32532, "parse_rate": 0.9842342342342343, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:41:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1343.6 tokens/s, Avg generation throughput: 2901.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.4%, Prefix cache hit rate: 0.4%
progress {"time": "08:42:59", "elapsed_min": 30.0, "submitted": 1677, "finished": 1421, "in_flight": 256, "gen_tokens": 4297616, "tok_per_s": 2387, "chain_tokens_median": 1586.0, "chain_tokens_p90": 6046.0, "chain_tokens_max": 32532, "parse_rate": 0.9852216748768473, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:42:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4315.3 tokens/s, Avg generation throughput: 2051.3 tokens/s, Running: 126 reqs, Waiting: 128 reqs, GPU KV cache usage: 53.9%, Prefix cache hit rate: 0.6%
progress {"time": "08:43:59", "elapsed_min": 31.0, "submitted": 1705, "finished": 1449, "in_flight": 256, "gen_tokens": 4431351, "tok_per_s": 2382, "chain_tokens_median": 1596.0, "chain_tokens_p90": 6065.200000000001, "chain_tokens_max": 32532, "parse_rate": 0.9841269841269841, "looped": 23, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:43:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1684.6 tokens/s, Avg generation throughput: 2676.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 52.3%, Prefix cache hit rate: 0.9%
progress {"time": "08:44:59", "elapsed_min": 32.0, "submitted": 1740, "finished": 1484, "in_flight": 256, "gen_tokens": 4502360, "tok_per_s": 2345, "chain_tokens_median": 1575.5, "chain_tokens_p90": 6044.1, "chain_tokens_max": 32532, "parse_rate": 0.9845013477088949, "looped": 23, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:44:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1733.2 tokens/s, Avg generation throughput: 2661.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 54.8%, Prefix cache hit rate: 1.3%
progress {"time": "08:45:59", "elapsed_min": 33.0, "submitted": 1821, "finished": 1565, "in_flight": 256, "gen_tokens": 4724627, "tok_per_s": 2386, "chain_tokens_median": 1576.0, "chain_tokens_p90": 5990.6, "chain_tokens_max": 32532, "parse_rate": 0.984664536741214, "looped": 24, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:45:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3980.6 tokens/s, Avg generation throughput: 1996.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 54.1%, Prefix cache hit rate: 1.8%
progress {"time": "08:46:59", "elapsed_min": 34.0, "submitted": 1839, "finished": 1583, "in_flight": 256, "gen_tokens": 4787783, "tok_per_s": 2347, "chain_tokens_median": 1586.0, "chain_tokens_p90": 6044.4, "chain_tokens_max": 32532, "parse_rate": 0.9848389134554643, "looped": 24, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:46:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1468.1 tokens/s, Avg generation throughput: 2728.7 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 57.2%, Prefix cache hit rate: 1.9%
progress {"time": "08:47:59", "elapsed_min": 35.0, "submitted": 1880, "finished": 1624, "in_flight": 256, "gen_tokens": 4897258, "tok_per_s": 2332, "chain_tokens_median": 1576.5, "chain_tokens_p90": 5990.7, "chain_tokens_max": 32532, "parse_rate": 0.9846059113300493, "looped": 25, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:47:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1896.3 tokens/s, Avg generation throughput: 2524.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 58.3%, Prefix cache hit rate: 2.3%
progress {"time": "08:48:59", "elapsed_min": 36.0, "submitted": 1950, "finished": 1694, "in_flight": 256, "gen_tokens": 5050366, "tok_per_s": 2338, "chain_tokens_median": 1575.5, "chain_tokens_p90": 5961.4, "chain_tokens_max": 32532, "parse_rate": 0.9852420306965761, "looped": 25, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:48:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3216.4 tokens/s, Avg generation throughput: 2126.8 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 60.4%, Prefix cache hit rate: 2.5%
progress {"time": "08:49:59", "elapsed_min": 37.0, "submitted": 1971, "finished": 1715, "in_flight": 256, "gen_tokens": 5128978, "tok_per_s": 2310, "chain_tokens_median": 1577.0, "chain_tokens_p90": 5955.200000000002, "chain_tokens_max": 32594, "parse_rate": 0.9848396501457726, "looped": 26, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:49:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1338.6 tokens/s, Avg generation throughput: 2598.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 61.8%, Prefix cache hit rate: 2.5%
progress {"time": "08:50:59", "elapsed_min": 38.0, "submitted": 2016, "finished": 1760, "in_flight": 256, "gen_tokens": 5314684, "tok_per_s": 2331, "chain_tokens_median": 1593.0, "chain_tokens_p90": 5924.600000000004, "chain_tokens_max": 32601, "parse_rate": 0.9835227272727273, "looped": 29, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:50:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2103.3 tokens/s, Avg generation throughput: 2403.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.3%, Prefix cache hit rate: 2.6%
progress {"time": "08:51:59", "elapsed_min": 39.0, "submitted": 2079, "finished": 1823, "in_flight": 256, "gen_tokens": 5527220, "tok_per_s": 2362, "chain_tokens_median": 1575.0, "chain_tokens_p90": 5888.799999999999, "chain_tokens_max": 32609, "parse_rate": 0.9824465167306637, "looped": 32, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:51:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2570.5 tokens/s, Avg generation throughput: 2290.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 59.3%, Prefix cache hit rate: 2.8%
progress {"time": "08:52:59", "elapsed_min": 40.0, "submitted": 2099, "finished": 1843, "in_flight": 256, "gen_tokens": 5609243, "tok_per_s": 2337, "chain_tokens_median": 1582.0, "chain_tokens_p90": 5918.599999999999, "chain_tokens_max": 32609, "parse_rate": 0.9820944112859469, "looped": 33, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:52:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1284.9 tokens/s, Avg generation throughput: 2675.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.2%, Prefix cache hit rate: 2.5%
progress {"time": "08:53:59", "elapsed_min": 41.0, "submitted": 2114, "finished": 1858, "in_flight": 256, "gen_tokens": 5695518, "tok_per_s": 2315, "chain_tokens_median": 1572.0, "chain_tokens_p90": 5951.599999999999, "chain_tokens_max": 32609, "parse_rate": 0.9811625403659849, "looped": 35, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:53:59 [loggers.py:335] Engine 000: Avg prompt throughput: 768.6 tokens/s, Avg generation throughput: 2793.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 59.4%, Prefix cache hit rate: 2.5%
progress {"time": "08:54:59", "elapsed_min": 42.0, "submitted": 2185, "finished": 1929, "in_flight": 256, "gen_tokens": 5907026, "tok_per_s": 2344, "chain_tokens_median": 1577.0, "chain_tokens_p90": 5908.400000000001, "chain_tokens_max": 32609, "parse_rate": 0.9813374805598756, "looped": 36, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:54:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3748.3 tokens/s, Avg generation throughput: 1960.4 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 59.0%, Prefix cache hit rate: 2.1%
progress {"time": "08:55:59", "elapsed_min": 43.0, "submitted": 2225, "finished": 1969, "in_flight": 256, "gen_tokens": 6039758, "tok_per_s": 2341, "chain_tokens_median": 1599.0, "chain_tokens_p90": 5908.400000000001, "chain_tokens_max": 32609, "parse_rate": 0.9812087353986796, "looped": 36, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:55:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2863.8 tokens/s, Avg generation throughput: 2279.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 61.4%, Prefix cache hit rate: 1.6%
progress {"time": "08:56:59", "elapsed_min": 44.0, "submitted": 2237, "finished": 1981, "in_flight": 256, "gen_tokens": 6132150, "tok_per_s": 2323, "chain_tokens_median": 1606.0, "chain_tokens_p90": 5922.0, "chain_tokens_max": 32609, "parse_rate": 0.9803129732458354, "looped": 38, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:56:59 [loggers.py:335] Engine 000: Avg prompt throughput: 825.5 tokens/s, Avg generation throughput: 2782.7 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 61.6%, Prefix cache hit rate: 1.4%
progress {"time": "08:57:59", "elapsed_min": 45.0, "submitted": 2277, "finished": 2021, "in_flight": 256, "gen_tokens": 6248411, "tok_per_s": 2314, "chain_tokens_median": 1602.0, "chain_tokens_p90": 5884.0, "chain_tokens_max": 32609, "parse_rate": 0.9802078179119248, "looped": 39, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:57:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1960.2 tokens/s, Avg generation throughput: 2397.4 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 63.4%, Prefix cache hit rate: 1.3%
progress {"time": "08:58:59", "elapsed_min": 46.0, "submitted": 2340, "finished": 2084, "in_flight": 256, "gen_tokens": 6434665, "tok_per_s": 2331, "chain_tokens_median": 1609.5, "chain_tokens_p90": 5872.0, "chain_tokens_max": 32609, "parse_rate": 0.9803262955854126, "looped": 40, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:58:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3265.1 tokens/s, Avg generation throughput: 1975.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 64.7%, Prefix cache hit rate: 0.8%
progress {"time": "08:59:59", "elapsed_min": 47.0, "submitted": 2360, "finished": 2104, "in_flight": 256, "gen_tokens": 6538259, "tok_per_s": 2318, "chain_tokens_median": 1615.5, "chain_tokens_p90": 5874.1, "chain_tokens_max": 32609, "parse_rate": 0.9795627376425855, "looped": 42, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 08:59:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1057.3 tokens/s, Avg generation throughput: 2620.2 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 63.7%, Prefix cache hit rate: 0.8%
progress {"time": "09:00:59", "elapsed_min": 48.0, "submitted": 2379, "finished": 2123, "in_flight": 256, "gen_tokens": 6641160, "tok_per_s": 2306, "chain_tokens_median": 1621.0, "chain_tokens_p90": 5882.2, "chain_tokens_max": 32609, "parse_rate": 0.9788035798398492, "looped": 44, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:00:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1096.5 tokens/s, Avg generation throughput: 2624.4 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.9%, Prefix cache hit rate: 0.5%
progress {"time": "09:01:59", "elapsed_min": 49.0, "submitted": 2433, "finished": 2177, "in_flight": 256, "gen_tokens": 6801584, "tok_per_s": 2313, "chain_tokens_median": 1625.0, "chain_tokens_p90": 5878.6, "chain_tokens_max": 32609, "parse_rate": 0.979329352319706, "looped": 44, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:01:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3335.1 tokens/s, Avg generation throughput: 2103.5 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 63.6%, Prefix cache hit rate: 0.3%
progress {"time": "09:02:59", "elapsed_min": 50.0, "submitted": 2483, "finished": 2227, "in_flight": 256, "gen_tokens": 7056645, "tok_per_s": 2352, "chain_tokens_median": 1632.0, "chain_tokens_p90": 5944.400000000001, "chain_tokens_max": 32609, "parse_rate": 0.9784463403682083, "looped": 47, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:02:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3284.0 tokens/s, Avg generation throughput: 2125.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 59.7%, Prefix cache hit rate: 0.2%
progress {"time": "09:03:59", "elapsed_min": 51.0, "submitted": 2502, "finished": 2246, "in_flight": 256, "gen_tokens": 7185483, "tok_per_s": 2348, "chain_tokens_median": 1628.0, "chain_tokens_p90": 5954.0, "chain_tokens_max": 32609, "parse_rate": 0.9772929652715939, "looped": 50, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:03:59 [loggers.py:335] Engine 000: Avg prompt throughput: 839.8 tokens/s, Avg generation throughput: 2825.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.4%, Prefix cache hit rate: 0.2%
progress {"time": "09:04:59", "elapsed_min": 52.0, "submitted": 2542, "finished": 2286, "in_flight": 256, "gen_tokens": 7353109, "tok_per_s": 2357, "chain_tokens_median": 1631.5, "chain_tokens_p90": 5954.0, "chain_tokens_max": 32609, "parse_rate": 0.9768153980752406, "looped": 52, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:04:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2328.6 tokens/s, Avg generation throughput: 2463.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 56.8%, Prefix cache hit rate: 0.2%
progress {"time": "09:05:59", "elapsed_min": 53.0, "submitted": 2599, "finished": 2343, "in_flight": 256, "gen_tokens": 7506482, "tok_per_s": 2360, "chain_tokens_median": 1657.0, "chain_tokens_p90": 5902.000000000003, "chain_tokens_max": 32609, "parse_rate": 0.9773794280836534, "looped": 52, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:05:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3190.8 tokens/s, Avg generation throughput: 2229.6 tokens/s, Running: 127 reqs, Waiting: 128 reqs, GPU KV cache usage: 58.6%, Prefix cache hit rate: 0.0%
progress {"time": "09:06:59", "elapsed_min": 54.0, "submitted": 2630, "finished": 2374, "in_flight": 256, "gen_tokens": 7690019, "tok_per_s": 2373, "chain_tokens_median": 1682.5, "chain_tokens_p90": 5916.900000000005, "chain_tokens_max": 32609, "parse_rate": 0.9764111204717776, "looped": 55, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:06:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1926.2 tokens/s, Avg generation throughput: 2595.8 tokens/s, Running: 127 reqs, Waiting: 128 reqs, GPU KV cache usage: 56.8%, Prefix cache hit rate: 0.0%
progress {"time": "09:07:59", "elapsed_min": 55.0, "submitted": 2659, "finished": 2403, "in_flight": 256, "gen_tokens": 7773368, "tok_per_s": 2355, "chain_tokens_median": 1720.0, "chain_tokens_p90": 5889.000000000001, "chain_tokens_max": 32609, "parse_rate": 0.976695796920516, "looped": 55, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:07:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1874.4 tokens/s, Avg generation throughput: 2580.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 60.4%, Prefix cache hit rate: 0.0%
progress {"time": "09:08:59", "elapsed_min": 56.0, "submitted": 2717, "finished": 2461, "in_flight": 256, "gen_tokens": 7896972, "tok_per_s": 2350, "chain_tokens_median": 1686.0, "chain_tokens_p90": 5872.0, "chain_tokens_max": 32609, "parse_rate": 0.9772450223486387, "looped": 55, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:08:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2715.0 tokens/s, Avg generation throughput: 2276.9 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 60.0%, Prefix cache hit rate: 0.0%
progress {"time": "09:09:59", "elapsed_min": 57.0, "submitted": 2760, "finished": 2504, "in_flight": 256, "gen_tokens": 8109477, "tok_per_s": 2371, "chain_tokens_median": 1723.5, "chain_tokens_p90": 5881.300000000003, "chain_tokens_max": 32609, "parse_rate": 0.9764376996805112, "looped": 58, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:09:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2452.9 tokens/s, Avg generation throughput: 2431.0 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 57.1%, Prefix cache hit rate: 0.0%
progress {"time": "09:10:59", "elapsed_min": 58.0, "submitted": 2783, "finished": 2527, "in_flight": 256, "gen_tokens": 8237496, "tok_per_s": 2367, "chain_tokens_median": 1745.0, "chain_tokens_p90": 5884.4, "chain_tokens_max": 32609, "parse_rate": 0.9758607043925603, "looped": 60, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:10:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1453.5 tokens/s, Avg generation throughput: 2728.7 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 57.3%, Prefix cache hit rate: 0.0%
progress {"time": "09:11:59", "elapsed_min": 59.0, "submitted": 2845, "finished": 2589, "in_flight": 256, "gen_tokens": 8485585, "tok_per_s": 2397, "chain_tokens_median": 1749.0, "chain_tokens_p90": 5885.0, "chain_tokens_max": 32609, "parse_rate": 0.9752800308999614, "looped": 63, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:11:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3196.2 tokens/s, Avg generation throughput: 2303.9 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 51.8%, Prefix cache hit rate: 0.0%
progress {"time": "09:12:59", "elapsed_min": 60.0, "submitted": 2893, "finished": 2637, "in_flight": 256, "gen_tokens": 8674516, "tok_per_s": 2409, "chain_tokens_median": 1790.0, "chain_tokens_p90": 5887.0, "chain_tokens_max": 32609, "parse_rate": 0.975350777398559, "looped": 64, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:12:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3230.1 tokens/s, Avg generation throughput: 2458.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 51.6%, Prefix cache hit rate: 0.0%
progress {"time": "09:13:59", "elapsed_min": 61.0, "submitted": 2926, "finished": 2670, "in_flight": 256, "gen_tokens": 8825417, "tok_per_s": 2411, "chain_tokens_median": 1816.5, "chain_tokens_p90": 5885.0, "chain_tokens_max": 32609, "parse_rate": 0.9749063670411985, "looped": 66, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:13:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1973.7 tokens/s, Avg generation throughput: 2781.3 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 50.9%, Prefix cache hit rate: 0.0%
progress {"time": "09:14:59", "elapsed_min": 62.0, "submitted": 3003, "finished": 2747, "in_flight": 256, "gen_tokens": 9134214, "tok_per_s": 2455, "chain_tokens_median": 1848.0, "chain_tokens_p90": 5872.0, "chain_tokens_max": 32609, "parse_rate": 0.9741536221332363, "looped": 70, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:14:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3829.7 tokens/s, Avg generation throughput: 2343.8 tokens/s, Running: 128 reqs, Waiting: 127 reqs, GPU KV cache usage: 46.6%, Prefix cache hit rate: 0.0%
progress {"time": "09:15:59", "elapsed_min": 63.0, "submitted": 3034, "finished": 2778, "in_flight": 256, "gen_tokens": 9239868, "tok_per_s": 2444, "chain_tokens_median": 1850.0, "chain_tokens_p90": 5853.800000000005, "chain_tokens_max": 32609, "parse_rate": 0.9740820734341252, "looped": 71, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:15:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2049.3 tokens/s, Avg generation throughput: 2772.1 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 47.1%, Prefix cache hit rate: 0.0%
progress {"time": "09:16:59", "elapsed_min": 64.0, "submitted": 3094, "finished": 2838, "in_flight": 256, "gen_tokens": 9417973, "tok_per_s": 2452, "chain_tokens_median": 1850.0, "chain_tokens_p90": 5835.200000000001, "chain_tokens_max": 32609, "parse_rate": 0.974277660324172, "looped": 72, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:16:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3508.2 tokens/s, Avg generation throughput: 2427.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 47.5%, Prefix cache hit rate: 0.1%
progress {"time": "09:17:59", "elapsed_min": 65.0, "submitted": 3145, "finished": 2889, "in_flight": 256, "gen_tokens": 9526840, "tok_per_s": 2443, "chain_tokens_median": 1848.0, "chain_tokens_p90": 5806.400000000007, "chain_tokens_max": 32609, "parse_rate": 0.9747317410868813, "looped": 72, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:17:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2765.2 tokens/s, Avg generation throughput: 2586.0 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 50.9%, Prefix cache hit rate: 0.4%
progress {"time": "09:18:59", "elapsed_min": 66.0, "submitted": 3193, "finished": 2937, "in_flight": 256, "gen_tokens": 9704069, "tok_per_s": 2450, "chain_tokens_median": 1848.0, "chain_tokens_p90": 5796.200000000001, "chain_tokens_max": 32609, "parse_rate": 0.9744637385086823, "looped": 74, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:18:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2828.7 tokens/s, Avg generation throughput: 2432.2 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 49.3%, Prefix cache hit rate: 1.0%
progress {"time": "09:19:59", "elapsed_min": 67.0, "submitted": 3268, "finished": 3012, "in_flight": 256, "gen_tokens": 9891835, "tok_per_s": 2461, "chain_tokens_median": 1814.0, "chain_tokens_p90": 5773.1, "chain_tokens_max": 32609, "parse_rate": 0.9744355909694555, "looped": 76, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:19:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2870.7 tokens/s, Avg generation throughput: 2446.8 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 48.0%, Prefix cache hit rate: 2.0%
progress {"time": "09:20:59", "elapsed_min": 68.0, "submitted": 3303, "finished": 3047, "in_flight": 256, "gen_tokens": 10060251, "tok_per_s": 2466, "chain_tokens_median": 1810.0, "chain_tokens_p90": 5782.6, "chain_tokens_max": 32609, "parse_rate": 0.9737446668854611, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:20:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1999.5 tokens/s, Avg generation throughput: 2791.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 47.0%, Prefix cache hit rate: 2.1%
progress {"time": "09:21:59", "elapsed_min": 69.0, "submitted": 3375, "finished": 3119, "in_flight": 256, "gen_tokens": 10205474, "tok_per_s": 2465, "chain_tokens_median": 1777.0, "chain_tokens_p90": 5761.800000000001, "chain_tokens_max": 32609, "parse_rate": 0.9743507534466175, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:21:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3165.8 tokens/s, Avg generation throughput: 2510.6 tokens/s, Running: 128 reqs, Waiting: 128 reqs, GPU KV cache usage: 44.6%, Prefix cache hit rate: 2.3%
progress {"time": "09:22:59", "elapsed_min": 70.0, "submitted": 3393, "finished": 3169, "in_flight": 224, "gen_tokens": 10346752, "tok_per_s": 2463, "chain_tokens_median": 1781.0, "chain_tokens_p90": 5761.800000000001, "chain_tokens_max": 32609, "parse_rate": 0.9747554433575261, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:22:59 [loggers.py:335] Engine 000: Avg prompt throughput: 3240.5 tokens/s, Avg generation throughput: 2577.3 tokens/s, Running: 128 reqs, Waiting: 95 reqs, GPU KV cache usage: 47.1%, Prefix cache hit rate: 3.1%
progress {"time": "09:23:59", "elapsed_min": 71.0, "submitted": 3393, "finished": 3258, "in_flight": 135, "gen_tokens": 10553734, "tok_per_s": 2477, "chain_tokens_median": 1752.0, "chain_tokens_p90": 5756.100000000001, "chain_tokens_max": 32609, "parse_rate": 0.9754450583179864, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:23:59 [loggers.py:335] Engine 000: Avg prompt throughput: 4118.8 tokens/s, Avg generation throughput: 2313.4 tokens/s, Running: 126 reqs, Waiting: 9 reqs, GPU KV cache usage: 48.4%, Prefix cache hit rate: 3.2%
progress {"time": "09:24:59", "elapsed_min": 72.0, "submitted": 3393, "finished": 3310, "in_flight": 83, "gen_tokens": 10687462, "tok_per_s": 2474, "chain_tokens_median": 1761.5, "chain_tokens_p90": 5745.2, "chain_tokens_max": 32609, "parse_rate": 0.9758308157099698, "looped": 79, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:24:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2549.9 tokens/s, Avg generation throughput: 2266.4 tokens/s, Running: 83 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.6%, Prefix cache hit rate: 3.4%
progress {"time": "09:25:59", "elapsed_min": 73.0, "submitted": 3393, "finished": 3342, "in_flight": 51, "gen_tokens": 10936272, "tok_per_s": 2497, "chain_tokens_median": 1809.5, "chain_tokens_p90": 5778.5, "chain_tokens_max": 32609, "parse_rate": 0.9751645721125075, "looped": 82, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:25:59 [loggers.py:335] Engine 000: Avg prompt throughput: 2774.6 tokens/s, Avg generation throughput: 1771.5 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.3%, Prefix cache hit rate: 3.1%
progress {"time": "09:26:59", "elapsed_min": 74.0, "submitted": 3393, "finished": 3362, "in_flight": 31, "gen_tokens": 11140853, "tok_per_s": 2509, "chain_tokens_median": 1825.5, "chain_tokens_p90": 5845.300000000001, "chain_tokens_max": 32609, "parse_rate": 0.97441998810232, "looped": 85, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:26:59 [loggers.py:335] Engine 000: Avg prompt throughput: 1923.5 tokens/s, Avg generation throughput: 1561.9 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 3.0%
progress {"time": "09:27:59", "elapsed_min": 75.0, "submitted": 3393, "finished": 3367, "in_flight": 26, "gen_tokens": 11209436, "tok_per_s": 2491, "chain_tokens_median": 1836.0, "chain_tokens_p90": 5878.6, "chain_tokens_max": 32609, "parse_rate": 0.9741609741609741, "looped": 86, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:27:59 [loggers.py:335] Engine 000: Avg prompt throughput: 619.2 tokens/s, Avg generation throughput: 1482.4 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 2.9%
progress {"time": "09:28:59", "elapsed_min": 76.0, "submitted": 3393, "finished": 3371, "in_flight": 22, "gen_tokens": 11300106, "tok_per_s": 2478, "chain_tokens_median": 1848.0, "chain_tokens_p90": 5890.0, "chain_tokens_max": 32609, "parse_rate": 0.9735983387718777, "looped": 88, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:28:59 [loggers.py:335] Engine 000: Avg prompt throughput: 447.3 tokens/s, Avg generation throughput: 1288.2 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 2.9%
progress {"time": "09:29:59", "elapsed_min": 77.0, "submitted": 3393, "finished": 3373, "in_flight": 20, "gen_tokens": 11364874, "tok_per_s": 2460, "chain_tokens_median": 1849.0, "chain_tokens_p90": 5891.6, "chain_tokens_max": 32609, "parse_rate": 0.9730210495108212, "looped": 90, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:29:59 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1163.2 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 2.9%
progress {"time": "09:30:59", "elapsed_min": 78.0, "submitted": 3393, "finished": 3374, "in_flight": 19, "gen_tokens": 11397252, "tok_per_s": 2435, "chain_tokens_median": 1849.5, "chain_tokens_p90": 5901.100000000004, "chain_tokens_max": 32609, "parse_rate": 0.972732661529342, "looped": 91, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:30:59 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1058.4 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.1%, Prefix cache hit rate: 2.9%
progress {"time": "09:31:59", "elapsed_min": 79.0, "submitted": 3393, "finished": 3380, "in_flight": 13, "gen_tokens": 11589503, "tok_per_s": 2445, "chain_tokens_median": 1857.5, "chain_tokens_p90": 5960.2, "chain_tokens_max": 32609, "parse_rate": 0.9710059171597634, "looped": 97, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:31:59 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 952.6 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 2.9%
progress {"time": "09:33:00", "elapsed_min": 80.0, "submitted": 3393, "finished": 3381, "in_flight": 12, "gen_tokens": 11621729, "tok_per_s": 2421, "chain_tokens_median": 1860.0, "chain_tokens_p90": 5962.0, "chain_tokens_max": 32609, "parse_rate": 0.9707187222715173, "looped": 98, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:33:00 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 789.7 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.0%, Prefix cache hit rate: 2.9%
progress {"time": "09:34:00", "elapsed_min": 81.0, "submitted": 3393, "finished": 3387, "in_flight": 6, "gen_tokens": 11815954, "tok_per_s": 2431, "chain_tokens_median": 1868.0, "chain_tokens_p90": 5985.200000000001, "chain_tokens_max": 32646, "parse_rate": 0.9689991142604074, "looped": 104, "minutes_to_deadline": Infinity}
(MainProcess pid=52652) INFO 10-31 09:34:00 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 633.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 2.9%
reason joint: {"submitted": 3393, "finished": 3393, "gen_tokens": 12010760, "t0": 1793434379.7662153, "seconds": 4885.523628473282, "tok_per_s": 2458.438626721644}
reason joint: post-chain readout 5847 taken at generation, 0 by prefill
reason joint: 5957 items, post-readout in 4886s total
latency joint: {"readout_ms": {"median": 40.53856432437897, "p90": 74.21627789735797, "n": 100}, "reasoning_ms": {"median": 11583.74154753983, "p90": 55556.02454841137, "n": 20}}
(MainProcess pid=52652) INFO 10-31 09:44:46 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=52652) INFO 10-31 09:44:46 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=52652) INFO 10-31 09:44:46 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=52921) INFO 10-31 09:44:46 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=52921) INFO 10-31 09:44:46 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=52921) INFO 10-31 09:44:46 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=52921) INFO 10-31 09:44:46 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=52652) INFO 10-31 09:44:49 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=52652) INFO 10-31 09:44:49 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=52652) INFO 10-31 09:44:49 [core_client.py:777] [shutdown] MPClient: complete
{"time": "2026-10-31T09:44:52Z", "event": "done", "stage": "eval_joint", "hours_left": -0.25}
{"time": "2026-10-31T09:45:14Z", "event": "start", "stage": "eval_gold", "hours_left": -0.26}
(MainProcess pid=57495) INFO 10-31 09:45:28 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM
(MainProcess pid=57495) INFO 10-31 09:45:28 [model.py:2152] Using max model len 32768
(MainProcess pid=57495) INFO 10-31 09:45:28 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=57495) INFO 10-31 09:45:28 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled
(MainProcess pid=57495) INFO 10-31 09:45:28 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=57770) INFO 10-31 09:45:34 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/gold_only_merged', speculative_config=None, tokenizer='checkpoints/gold_only_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/gold_only_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=57770) INFO 10-31 09:45:36 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_cded85161b374b0f9cd4fead486e56d8 backend=nccl
(EngineCore pid=57770) INFO 10-31 09:45:36 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=57770) INFO 10-31 09:45:36 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=57770) INFO 10-31 09:45:37 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=57770) INFO 10-31 09:45:37 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=57770) INFO 10-31 09:45:39 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=57770) INFO 10-31 09:45:39 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=57770) INFO 10-31 09:45:39 [base_loader.py:82] Initializing model took 1.584827 seconds
(EngineCore pid=57770) INFO 10-31 09:45:39 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 162.14 GiB.
(EngineCore pid=57770) INFO 10-31 09:45:39 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.22it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.22it/s]
(EngineCore pid=57770) INFO 10-31 09:45:40 [default_loader.py:501] Loading weights took 0.84 seconds
(EngineCore pid=57770) INFO 10-31 09:45:42 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.588046 seconds
(EngineCore pid=57770) INFO 10-31 09:45:42 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=57770) INFO 10-31 09:45:42 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=57770) INFO 10-31 09:45:42 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=57770) INFO 10-31 09:45:42 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=57770) INFO 10-31 09:45:42 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=57770) INFO 10-31 09:45:43 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=57770) INFO 10-31 09:45:48 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/8a9d0bdb28/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=57770) INFO 10-31 09:45:48 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.48 s
(EngineCore pid=57770) INFO 10-31 09:46:06 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 17.35 s
(EngineCore pid=57770) INFO 10-31 09:46:08 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20433929 bytes total
(EngineCore pid=57770) INFO 10-31 09:46:08 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/a412c7d5e23763e33eecbcc5f7b227476f13259ff5446f4d73fb6b16a8691e68/rank_0_0/model
(EngineCore pid=57770) INFO 10-31 09:46:08 [monitor.py:62] [backbone] torch.compile took 25.85 s in total
(EngineCore pid=57770) INFO 10-31 09:47:37 [monitor.py:91] [backbone] Initial profiling/warmup run took 88.35 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:32, 1.03it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:01<00:25, 1.27it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:02<00:20, 1.55it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:02<00:15, 2.05it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:02<00:14, 2.05it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:03<00:11, 2.51it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:03<00:11, 2.41it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:03<00:09, 2.81it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:04<00:10, 2.54it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:04<00:10, 2.43it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:04<00:08, 2.78it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:05<00:08, 2.64it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:05<00:07, 3.14it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:05<00:05, 3.61it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:06<00:05, 3.41it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:06<00:04, 4.13it/s]Capturing CUDA graphs (PIECEWISE): 49%|████▊ | 17/35 [00:07<00:09, 1.96it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:07<00:07, 2.33it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:07<00:03, 3.92it/s]Capturing CUDA graphs (PIECEWISE): 60%|██████ | 21/35 [00:07<00:03, 3.94it/s]Capturing CUDA graphs (PIECEWISE): 66%|██████▌ | 23/35 [00:07<00:02, 5.82it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:08<00:01, 8.83it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:08<00:00, 10.20it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:11<00:02, 2.04it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:11<00:01, 2.76it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:11<00:00, 3.71it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:11<00:00, 3.01it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 23.80it/s]
(EngineCore pid=57770) INFO 10-31 09:47:52 [model_runner.py:1127] Graph capturing finished in 13 secs, took 0.30 GiB
(EngineCore pid=57770) INFO 10-31 09:47:53 [gpu_worker.py:759] Available KV cache memory: 59.38 GiB
(EngineCore pid=57770) INFO 10-31 09:47:53 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=57770) INFO 10-31 09:47:53 [kv_cache_utils.py:2327] GPU KV cache size: 1,749,526 tokens, Maximum concurrency for 32,768 tokens per request: 53.39x
(EngineCore pid=57770) INFO 10-31 09:47:53 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=57770) INFO 10-31 09:47:53 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=57770) INFO 10-31 09:47:53 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=57770) INFO 10-31 09:47:53 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=57770) INFO 10-31 09:47:53 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=57770) INFO 10-31 09:47:54 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:12, 2.77it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:14, 2.30it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:12, 2.66it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:12, 2.51it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:11, 2.68it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:10, 2.82it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:09, 2.98it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:09, 2.99it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:10, 2.59it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:08, 2.88it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:04<00:09, 2.66it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:04<00:07, 3.11it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:07, 2.86it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:06, 3.42it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:04, 4.11it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:05<00:05, 3.60it/s]Capturing CUDA graphs (PIECEWISE): 54%|█████▍ | 19/35 [00:05<00:02, 6.79it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:05<00:01, 9.85it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:05<00:01, 10.45it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:06<00:00, 9.58it/s]Capturing CUDA graphs (PIECEWISE): 83%|████████▎ | 29/35 [00:06<00:00, 12.37it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:06<00:00, 14.68it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:06<00:00, 12.17it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:06<00:00, 5.29it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 28.04it/s]Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 28.70it/s]Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 29.04it/s]Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 18.63it/s]Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 19.84it/s]Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 21.96it/s]Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 22.65it/s]
(EngineCore pid=57770) INFO 10-31 09:48:21 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.16 GiB
(EngineCore pid=57770) INFO 10-31 09:48:21 [gpu_worker.py:971] CUDA graph pool memory: 0.16 GiB (actual), 0.32 GiB (estimated), difference: 0.16 GiB (103.8%).
(EngineCore pid=57770) INFO 10-31 09:48:21 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 10.39 GiB for consumed memory (weights + non-torch), 1.55 GiB for peak activation, and 0.16 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=63430039962` (59.07 GiB) to fit into requested memory, or `--kv-cache-memory=71496105984` (66.59 GiB) to fully utilize gpu memory. Current kv cache memory in use is 59.38 GiB.
(EngineCore pid=57770) INFO 10-31 09:48:22 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=57770) INFO 10-31 09:48:23 [core.py:395] init engine (profile, create kv cache, warmup model) took 160.87 s (compilation: 25.85 s)
(EngineCore pid=57770) INFO 10-31 09:48:23 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=57495) WARNING 10-31 09:48:24 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=57770) WARNING 10-31 09:48:24 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout gold_only: 5957 items in 114s
(MainProcess pid=57495) INFO 10-31 09:50:16 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=57495) INFO 10-31 09:50:16 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=57495) INFO 10-31 09:50:16 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=57770) INFO 10-31 09:50:16 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=57770) INFO 10-31 09:50:16 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=57770) INFO 10-31 09:50:16 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=57770) INFO 10-31 09:50:16 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=57495) INFO 10-31 09:50:19 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=57495) INFO 10-31 09:50:19 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=57495) INFO 10-31 09:50:19 [core_client.py:777] [shutdown] MPClient: complete
{"time": "2026-10-31T09:50:22Z", "event": "done", "stage": "eval_gold", "hours_left": -0.34}
{"time": "2026-10-31T09:50:43Z", "event": "start", "stage": "eval_votes", "hours_left": -0.35}
(MainProcess pid=59443) INFO 10-31 09:50:55 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM
(MainProcess pid=59443) INFO 10-31 09:50:55 [model.py:2152] Using max model len 32768
(MainProcess pid=59443) INFO 10-31 09:50:55 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=59443) INFO 10-31 09:50:55 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled
(MainProcess pid=59443) INFO 10-31 09:50:56 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=59778) INFO 10-31 09:51:02 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/votes_only_merged', speculative_config=None, tokenizer='checkpoints/votes_only_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/votes_only_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=59778) INFO 10-31 09:51:03 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_1c8f0f7a83324ab2b1fef8874eca4bf3 backend=nccl
(EngineCore pid=59778) INFO 10-31 09:51:03 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=59778) INFO 10-31 09:51:03 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=59778) INFO 10-31 09:51:05 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=59778) INFO 10-31 09:51:05 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=59778) INFO 10-31 09:51:06 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=59778) INFO 10-31 09:51:06 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=59778) INFO 10-31 09:51:07 [base_loader.py:82] Initializing model took 1.759581 seconds
(EngineCore pid=59778) INFO 10-31 09:51:07 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 161.98 GiB.
(EngineCore pid=59778) INFO 10-31 09:51:07 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.03it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.03it/s]
(EngineCore pid=59778) INFO 10-31 09:51:08 [default_loader.py:501] Loading weights took 1.00 seconds
(EngineCore pid=59778) INFO 10-31 09:51:09 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.853671 seconds
(EngineCore pid=59778) INFO 10-31 09:51:09 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=59778) INFO 10-31 09:51:09 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=59778) INFO 10-31 09:51:09 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=59778) INFO 10-31 09:51:09 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=59778) INFO 10-31 09:51:09 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=59778) INFO 10-31 09:51:10 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=59778) INFO 10-31 09:51:16 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/3f1d4624f6/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=59778) INFO 10-31 09:51:16 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.43 s
(EngineCore pid=59778) INFO 10-31 09:51:34 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 18.10 s
(EngineCore pid=59778) INFO 10-31 09:51:37 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20433988 bytes total
(EngineCore pid=59778) INFO 10-31 09:51:37 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/3eb3c78fc507f33c9366ef8e9bbf26e7f9a0836847e49322673b158d219c79f3/rank_0_0/model
(EngineCore pid=59778) INFO 10-31 09:51:37 [monitor.py:62] [backbone] torch.compile took 26.51 s in total
(EngineCore pid=59778) INFO 10-31 09:53:03 [monitor.py:91] [backbone] Initial profiling/warmup run took 86.30 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:01<01:04, 1.91s/it]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:02<00:38, 1.18s/it]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:02<00:24, 1.33it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:03<00:17, 1.82it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:03<00:12, 2.32it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:03<00:10, 2.78it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:03<00:08, 3.21it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:03<00:07, 3.55it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:04<00:07, 3.52it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:04<00:06, 3.58it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:04<00:06, 3.66it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:04<00:06, 3.82it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:05<00:05, 3.98it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:05<00:04, 4.22it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:05<00:04, 4.71it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:05<00:03, 5.36it/s]Capturing CUDA graphs (PIECEWISE): 49%|████▊ | 17/35 [00:06<00:08, 2.24it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:06<00:06, 2.61it/s]Capturing CUDA graphs (PIECEWISE): 60%|██████ | 21/35 [00:07<00:02, 5.11it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:07<00:01, 7.73it/s]Capturing CUDA graphs (PIECEWISE): 77%|███████▋ | 27/35 [00:07<00:00, 10.31it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:07<00:00, 12.71it/s]Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 33/35 [00:07<00:00, 14.72it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:07<00:00, 12.19it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:07<00:00, 4.41it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 26.09it/s]
(EngineCore pid=59778) INFO 10-31 09:53:15 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.30 GiB
(EngineCore pid=59778) INFO 10-31 09:53:16 [gpu_worker.py:759] Available KV cache memory: 59.38 GiB
(EngineCore pid=59778) INFO 10-31 09:53:16 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=59778) INFO 10-31 09:53:16 [kv_cache_utils.py:2327] GPU KV cache size: 1,749,526 tokens, Maximum concurrency for 32,768 tokens per request: 53.39x
(EngineCore pid=59778) INFO 10-31 09:53:16 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=59778) INFO 10-31 09:53:16 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=59778) INFO 10-31 09:53:16 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=59778) INFO 10-31 09:53:16 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=59778) INFO 10-31 09:53:16 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=59778) INFO 10-31 09:53:17 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:10, 3.19it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.69it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:11, 2.68it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.74it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:11, 2.69it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:04<00:34, 1.18s/it]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:04<00:24, 1.16it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:05<00:17, 1.52it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:05<00:14, 1.84it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:05<00:11, 2.19it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:05<00:09, 2.56it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:06<00:07, 3.01it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:06<00:06, 3.51it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:06<00:05, 4.05it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:06<00:04, 4.73it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:06<00:03, 5.44it/s]Capturing CUDA graphs (PIECEWISE): 54%|█████▍ | 19/35 [00:06<00:01, 9.40it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:06<00:01, 12.65it/s]Capturing CUDA graphs (PIECEWISE): 71%|███████▏ | 25/35 [00:06<00:00, 15.26it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:07<00:00, 17.23it/s]Capturing CUDA graphs (PIECEWISE): 89%|████████▊ | 31/35 [00:07<00:00, 18.80it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:07<00:00, 19.93it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:07<00:00, 4.71it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 28.47it/s]Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 28.95it/s]Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 28.95it/s]Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 28.37it/s]Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 28.49it/s]Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 28.21it/s]Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 28.40it/s]
(EngineCore pid=59778) INFO 10-31 09:53:45 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.16 GiB
(EngineCore pid=59778) INFO 10-31 09:53:45 [gpu_worker.py:971] CUDA graph pool memory: 0.16 GiB (actual), 0.32 GiB (estimated), difference: 0.16 GiB (103.8%).
(EngineCore pid=59778) INFO 10-31 09:53:45 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 10.39 GiB for consumed memory (weights + non-torch), 1.55 GiB for peak activation, and 0.16 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=63430039962` (59.07 GiB) to fit into requested memory, or `--kv-cache-memory=71496105984` (66.59 GiB) to fully utilize gpu memory. Current kv cache memory in use is 59.38 GiB.
(EngineCore pid=59778) INFO 10-31 09:53:46 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=59778) INFO 10-31 09:53:46 [core.py:395] init engine (profile, create kv cache, warmup model) took 156.74 s (compilation: 26.51 s)
(EngineCore pid=59778) INFO 10-31 09:53:46 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=59443) WARNING 10-31 09:53:47 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=59778) WARNING 10-31 09:53:47 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout votes_only: 5957 items in 114s
(MainProcess pid=59443) INFO 10-31 09:55:40 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=59443) INFO 10-31 09:55:40 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=59443) INFO 10-31 09:55:40 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=59778) INFO 10-31 09:55:40 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=59778) INFO 10-31 09:55:40 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=59778) INFO 10-31 09:55:40 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=59778) INFO 10-31 09:55:40 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=59443) INFO 10-31 09:55:43 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=59443) INFO 10-31 09:55:43 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=59443) INFO 10-31 09:55:43 [core_client.py:777] [shutdown] MPClient: complete
{"time": "2026-10-31T09:55:45Z", "event": "done", "stage": "eval_votes", "hours_left": -0.43}
{"time": "2026-10-31T09:56:29Z", "event": "start", "stage": "metrics", "hours_left": -0.44}
{
"H1": {
"joint_minus_gold_only_acc": 0.000888888888888939,
"ci95": [
-0.0204666666666666,
0.022222222222222254
],
"confirmed": false
},
"H2": {
"joint_minus_gold_only": {
"delta": 0.011839698959607348,
"ci95": [
-0.0018633475228017975,
0.023158437117584887
],
"ok": false
},
"joint_minus_base": {
"delta": 0.010722106192774361,
"ci95": [
-0.003452832367084524,
0.024402413121459573
],
"ok": false
},
"confirmed": false
},
"H3": {
"joint_minus_base_reasoning_acc": 0.0,
"ci95": [
-0.008273612642410874,
0.007180264608599827
],
"confirmed": true
},
"H4": {
"gap_recovered_f0.3": 0.5265700483091791,
"readout_acc": 0.6945975744211687,
"reasoning_acc": 0.8467475192943771,
"confirmed": false,
"gap_recovered_f0.3_learned_router": 0.5132850241545892
}
}
{"time": "2026-10-31T09:57:19Z", "event": "done", "stage": "metrics", "hours_left": -0.46}
{"time": "2026-10-31T09:57:53Z", "event": "start", "stage": "report", "hours_left": -0.47}
# Reasoning decision model: results
## Hypotheses
| # | Result | Numbers | Confirmed |
|---|---|---|---|
| H1 | joint readout vs gold-only, in-distribution accuracy | +0.1 pts [-2.0, +2.2] | no |
| H2 | disagreement AUROC, joint vs others (need +0.03) | vs gold_only +0.012 [-0.002, +0.023]; vs base +0.011 [-0.003, +0.024] | no |
| H3 | joint reasoner vs base reasoner, all eval | +0.0 pts [-0.8, +0.7] | yes |
| H4 | gap recovered at f = 0.3 (need 80%) | 52.7% of 69.5 → 84.7; learned router 51.3% | no |
## Readout
Accuracy (%) after per-type temperatures fitted on the calibration set; ECE uses 15 bins.
| Readout | Procedural | tasksource | OpenDecisions | LocalLLaMA | LocalLLaMA (panel agrees) | OD Capability | OD Capability (raw) | In-dist ECE | In-dist NLL |
|---|---|---|---|---|---|---|---|---|---|
| base | 46.8 | 60.4 | 66.5 | 63.0 | 74.1 | 61.8 | 55.2 | 0.042 | 1.027 |
| gold_only | 72.4 | 65.8 | 72.1 | 67.5 | 76.7 | 66.0 | 65.1 | 0.032 | 0.774 |
| votes_only | 71.7 | 66.3 | 71.9 | 66.3 | 75.1 | 66.4 | 62.3 | 0.041 | 0.780 |
| joint | 71.9 | 66.4 | 72.2 | 66.5 | 76.3 | 66.7 | 64.6 | 0.031 | 0.757 |
Temperatures (choice / yes-no / score): base 1.71 / 2.41 / 1.28; gold_only 1.18 / 0.87 / 1.02; votes_only 1.60 / 1.08 / 1.48; joint 1.28 / 1.18 / 1.05
## Reasoning (one sample per item)
| Reasoner | Procedural | tasksource | OpenDecisions | LocalLLaMA | All | Median tokens | p90 tokens | Looped | Letter = readout after chain |
|---|---|---|---|---|---|---|---|---|---|
| base | 95.1 | 73.3 | 94.9 | 71.0 | 83.9 | 1626 | 5737 | 1.0% | 98.9% |
| joint | 97.1 | 74.3 | 94.1 | 71.0 | 83.9 | 1412 | 5497 | 1.7% | 98.3% |
## Does the readout agree with its reasoning?
Agreement = the readout's answer equals the same model's thinking-mode answer (the base reasoner for the readout-only ablations). The thinking call never sees the readout's answer, so this is also the faithfulness check of explanation on demand.
| Readout | Agreement | Procedural | tasksource | OpenDecisions | LocalLLaMA | Disagreement AUROC | Fixable AUROC |
|---|---|---|---|---|---|---|---|
| base | 66.8 | 46.8 | 67.9 | 67.8 | 70.6 | 0.778 | 0.580 |
| gold_only | 71.6 | 71.6 | 72.8 | 73.3 | 69.0 | 0.777 | 0.489 |
| votes_only | 73.1 | 71.4 | 76.2 | 72.9 | 72.7 | 0.803 | 0.562 |
| joint | 74.4 | 72.2 | 75.8 | 74.1 | 75.0 | 0.789 | 0.526 |
## Routing
Route the lowest-margin share f of items to thinking mode (all eval sets pooled).
| Readout | Router | f=0 | 0.1 | 0.2 | 0.3 | 0.5 | 1.0 | tokens at 0.3 | random at 0.3 |
|---|---|---|---|---|---|---|---|---|---|
| base | margin | 62.7 | 66.9 | 70.8 | 74.4 | 79.3 | 84.3 | 1095 | 69.2 |
| base | learned | 62.7 | 67.3 | 71.1 | 74.5 | 79.5 | 84.3 | 1138 | 69.2 |
| gold_only | margin | 69.8 | 73.1 | 75.3 | 77.5 | 80.4 | 84.4 | 1134 | 74.2 |
| gold_only | learned | 69.8 | 72.7 | 75.1 | 76.8 | 80.4 | 84.4 | 1181 | 74.2 |
| votes_only | margin | 69.2 | 72.6 | 75.7 | 78.1 | 81.7 | 84.4 | 1177 | 73.8 |
| votes_only | learned | 69.2 | 72.9 | 75.5 | 78.2 | 81.7 | 84.4 | 1204 | 73.8 |
| joint | margin | 69.5 | 72.5 | 75.4 | 77.5 | 80.9 | 84.7 | 1244 | 74.0 |
| joint | learned | 69.5 | 73.0 | 75.0 | 77.3 | 81.1 | 84.7 | 1275 | 74.0 |
## The deployable system
Readout by default; items with the lowest readout margin go to thinking mode, whose probabilities are the readout after the chain (temperatures fitted on calibration chains only). All eval sets; OpenDecisions Capability with the official scorer.
| Model | Mode | Routed | Procedural | tasksource | OpenDecisions | LocalLLaMA | All | OD Capability | Mean generated tokens |
|---|---|---|---|---|---|---|---|---|---|
| base | readout only | 0.0% | 46.8 | 60.4 | 66.5 | 63.0 | 62.7 | 61.8 | 0 |
| base | routed | 10.0% | 58.6 | 64.3 | 71.1 | 65.0 | 66.9 | 67.2 | 367 |
| base | routed | 20.0% | 67.3 | 66.1 | 76.0 | 67.1 | 70.9 | 72.3 | 730 |
| base | routed | 30.0% | 79.3 | 68.1 | 80.4 | 68.1 | 74.5 | 76.5 | 1095 |
| base | routed | 50.0% | 90.9 | 71.3 | 87.2 | 69.6 | 79.5 | 82.8 | 1718 |
| base | thinking always | 100.0% | 96.0 | 74.0 | 95.4 | 71.7 | 84.5 | 84.4 | 2823 |
| joint | readout only | 0.0% | 71.9 | 66.4 | 72.2 | 66.5 | 69.5 | 66.7 | 0 |
| joint | routed | 10.0% | 77.8 | 68.3 | 75.9 | 68.1 | 72.5 | 69.8 | 429 |
| joint | routed | 20.0% | 82.1 | 70.1 | 79.9 | 69.8 | 75.4 | 73.6 | 829 |
| joint | routed | 30.0% | 86.5 | 71.5 | 82.9 | 70.4 | 77.5 | 77.1 | 1244 |
| joint | routed | 50.0% | 92.0 | 73.8 | 87.8 | 71.9 | 80.9 | 83.2 | 1946 |
| joint | thinking always | 100.0% | 97.5 | 74.5 | 94.6 | 72.5 | 84.7 | 88.1 | 2831 |
Post-chain temperatures (choice / yes-no / score): base 4.77 / 3.49 / 4.02; joint 4.08 / 2.94 / 3.65
## Latency (batch size 1)
| Model | Readout median | Readout p90 | Reasoning median | Reasoning p90 |
|---|---|---|---|---|
| base | 43 ms | 76 ms | 10.1 s | 41.8 s |
| joint | 41 ms | 74 ms | 11.6 s | 55.6 s |
## Run
- Throughput probe: 1957 generated tokens/s; chain tokens median 1878, p90 6414, max 32471; parse rate 99.2%.
- Generation: 3169 items, 12676 chains, 41.0M tokens; 2818 items with a correct chain; 154 chains looped to the context limit.
- Training joint: 2673 reasoner + 2996 readout examples, 8.7M tokens, 207/207 steps, 46 min (3137 tokens/s).
- Training gold_only: 0 reasoner + 3169 readout examples, 1.7M tokens, 37/37 steps, 11 min (2521 tokens/s).
- Training votes_only: 0 reasoner + 3169 readout examples, 1.7M tokens, 37/37 steps, 7 min (3898 tokens/s).
- Single seed (17); see results/DEVIATIONS.md for every change to the plan.
{"time": "2026-10-31T09:57:53Z", "event": "done", "stage": "report", "hours_left": -0.47}
{"time": "2026-10-31T09:58:38Z", "event": "finished", "stage": "all", "hours_left": -0.48}
RUN_ALL_EXIT 0
{"time": "2026-10-31T00:49:37Z", "event": "skip", "stage": "prep", "hours_left": 8.0}
{"time": "2026-10-31T00:49:37Z", "event": "start", "stage": "probe", "hours_left": 8.0}
(MainProcess pid=25496) INFO 10-31 00:49:49 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration
(MainProcess pid=25496) INFO 10-31 00:49:49 [model.py:2152] Using max model len 32768
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1166.68it/s]
(MainProcess pid=25496) INFO 10-31 00:49:51 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=25496) INFO 10-31 00:49:51 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.73it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 7.62it/s]
(MainProcess pid=25496) INFO 10-31 00:49:52 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=25496) WARNING 10-31 00:50:03 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
(EngineCore pid=25921) INFO 10-31 00:50:14 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=25921) INFO 10-31 00:50:15 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_1cdc2a918df5485caa3258a5a3b3664c backend=nccl
(EngineCore pid=25921) INFO 10-31 00:50:15 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=25921) INFO 10-31 00:50:15 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=25921) INFO 10-31 00:50:17 [model_runner.py:392] Loading model from scratch...
Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s]Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1077.81it/s]
(EngineCore pid=25921) INFO 10-31 00:50:18 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=25921) INFO 10-31 00:50:18 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=25921) INFO 10-31 00:50:18 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=25921) INFO 10-31 00:50:19 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=25921) INFO 10-31 00:50:19 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=25921) INFO 10-31 00:50:19 [base_loader.py:82] Initializing model took 2.131808 seconds
(EngineCore pid=25921) INFO 10-31 00:50:19 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 193.47 GiB.
(EngineCore pid=25921) INFO 10-31 00:50:19 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.67it/s]
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
(EngineCore pid=25921) WARNING 10-31 00:50:20 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.68it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.68it/s]
(EngineCore pid=25921) INFO 10-31 00:50:20 [default_loader.py:501] Loading weights took 0.75 seconds
(EngineCore pid=25921) INFO 10-31 00:50:21 [model_runner.py:424] Model loading took 8.56 GiB memory and 4.702506 seconds
(EngineCore pid=25921) INFO 10-31 00:50:21 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=25921) INFO 10-31 00:50:21 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=25921) INFO 10-31 00:50:21 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=25921) INFO 10-31 00:50:21 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=25921) INFO 10-31 00:50:21 [utils.py:347] Using LBNHC KV cache layout.
[transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning.
(MainProcess pid=25496) INFO 10-31 00:50:23 [base.py:280] Multi-modal warmup completed in 20.065s
(EngineCore pid=25921) INFO 10-31 00:50:29 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=25921) INFO 10-31 00:50:34 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=25921) INFO 10-31 00:50:39 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/c4af2daa17/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=25921) INFO 10-31 00:50:39 [backends.py:1166] [backbone] Dynamo bytecode transform time: 4.61 s
(EngineCore pid=25921) INFO 10-31 00:50:56 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 17.28 s
(EngineCore pid=25921) INFO 10-31 00:50:59 [backends.py:927] collected artifacts: 33 entries, 13 artifacts, 18863803 bytes total
(EngineCore pid=25921) INFO 10-31 00:50:59 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/cc0f0cae53dc3e1375a6d2ed4c88238808e3367e1d9abaa0419cc3d58ebc1b12/rank_0_0/model
(EngineCore pid=25921) INFO 10-31 00:50:59 [monitor.py:62] [backbone] torch.compile took 24.79 s in total
(EngineCore pid=25921) INFO 10-31 00:52:25 [monitor.py:91] [backbone] Initial profiling/warmup run took 85.73 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:30, 1.12it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:01<00:26, 1.22it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:18, 1.75it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:02<00:15, 2.03it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:02<00:13, 2.24it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:11, 2.63it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:03<00:10, 2.77it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:03<00:08, 3.12it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:08, 3.00it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:04<00:08, 2.95it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:04<00:07, 3.25it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:04<00:07, 3.28it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:06, 3.67it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:07<00:21, 1.02s/it]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:07<00:15, 1.31it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:08<00:11, 1.71it/s]Capturing CUDA graphs (PIECEWISE): 49%|████▊ | 17/35 [00:09<00:12, 1.39it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:09<00:09, 1.76it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:09<00:05, 2.81it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:09<00:03, 4.13it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:09<00:01, 5.73it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:09<00:01, 7.39it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:10<00:00, 8.35it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:10<00:00, 10.09it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:10<00:00, 11.78it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:10<00:00, 13.05it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:10<00:00, 3.31it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 23.26it/s]
(EngineCore pid=25921) INFO 10-31 00:52:39 [model_runner.py:1127] Graph capturing finished in 12 secs, took 0.28 GiB
(EngineCore pid=25921) INFO 10-31 00:52:39 [gpu_worker.py:759] Available KV cache memory: 58.36 GiB
(EngineCore pid=25921) INFO 10-31 00:52:39 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8960 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9040. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=25921) INFO 10-31 00:52:39 [kv_cache_utils.py:2327] GPU KV cache size: 1,719,607 tokens, Maximum concurrency for 32,768 tokens per request: 52.48x
(EngineCore pid=25921) INFO 10-31 00:52:39 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=25921) INFO 10-31 00:52:39 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=25921) INFO 10-31 00:52:39 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=25921) INFO 10-31 00:52:39 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=25921) INFO 10-31 00:52:39 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=25921) INFO 10-31 00:52:41 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)].
(EngineCore pid=25921) INFO 10-31 00:52:41 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/35 [00:00<?, ?it/s]Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/35 [00:00<00:11, 2.99it/s]Capturing CUDA graphs (PIECEWISE): 6%|▌ | 2/35 [00:00<00:12, 2.59it/s]Capturing CUDA graphs (PIECEWISE): 9%|▊ | 3/35 [00:01<00:12, 2.59it/s]Capturing CUDA graphs (PIECEWISE): 11%|█▏ | 4/35 [00:01<00:11, 2.65it/s]Capturing CUDA graphs (PIECEWISE): 14%|█▍ | 5/35 [00:01<00:10, 2.76it/s]Capturing CUDA graphs (PIECEWISE): 17%|█▋ | 6/35 [00:02<00:10, 2.83it/s]Capturing CUDA graphs (PIECEWISE): 20%|██ | 7/35 [00:02<00:09, 3.10it/s]Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 8/35 [00:02<00:08, 3.26it/s]Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 9/35 [00:03<00:08, 2.95it/s]Capturing CUDA graphs (PIECEWISE): 29%|██▊ | 10/35 [00:03<00:07, 3.18it/s]Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 11/35 [00:03<00:07, 3.20it/s]Capturing CUDA graphs (PIECEWISE): 34%|███▍ | 12/35 [00:03<00:06, 3.55it/s]Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 13/35 [00:04<00:06, 3.51it/s]Capturing CUDA graphs (PIECEWISE): 40%|████ | 14/35 [00:04<00:05, 3.98it/s]Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 15/35 [00:04<00:05, 4.00it/s]Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 16/35 [00:04<00:04, 4.43it/s]Capturing CUDA graphs (PIECEWISE): 51%|█████▏ | 18/35 [00:04<00:02, 6.62it/s]Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 20/35 [00:05<00:02, 7.38it/s]Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 22/35 [00:05<00:01, 9.07it/s]Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 24/35 [00:05<00:01, 10.69it/s]Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 26/35 [00:05<00:00, 11.95it/s]Capturing CUDA graphs (PIECEWISE): 80%|████████ | 28/35 [00:05<00:00, 11.12it/s]Capturing CUDA graphs (PIECEWISE): 86%|████████▌ | 30/35 [00:05<00:00, 12.50it/s]Capturing CUDA graphs (PIECEWISE): 91%|█████████▏| 32/35 [00:05<00:00, 13.45it/s]Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 34/35 [00:06<00:00, 11.06it/s]Capturing CUDA graphs (PIECEWISE): 100%|██████████| 35/35 [00:06<00:00, 5.58it/s]
Capturing CUDA graphs (FULL): 0%| | 0/19 [00:00<?, ?it/s]Capturing CUDA graphs (FULL): 16%|█▌ | 3/19 [00:00<00:00, 25.15it/s]Capturing CUDA graphs (FULL): 32%|███▏ | 6/19 [00:00<00:00, 25.44it/s]Capturing CUDA graphs (FULL): 47%|████▋ | 9/19 [00:00<00:00, 22.67it/s]Capturing CUDA graphs (FULL): 63%|██████▎ | 12/19 [00:00<00:00, 22.18it/s]Capturing CUDA graphs (FULL): 79%|███████▉ | 15/19 [00:00<00:00, 23.26it/s]Capturing CUDA graphs (FULL): 95%|█████████▍| 18/19 [00:00<00:00, 23.42it/s]Capturing CUDA graphs (FULL): 100%|██████████| 19/19 [00:00<00:00, 23.23it/s]
(EngineCore pid=25921) INFO 10-31 00:53:08 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.14 GiB
(EngineCore pid=25921) INFO 10-31 00:53:08 [gpu_worker.py:971] CUDA graph pool memory: 0.14 GiB (actual), 0.32 GiB (estimated), difference: 0.18 GiB (132.9%).
(EngineCore pid=25921) INFO 10-31 00:53:08 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.9, 71.33 GiB). Actual usage is 11.05 GiB for consumed memory (weights + non-torch), 1.92 GiB for peak activation, and 0.14 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=62358280090` (58.08 GiB) to fit into requested memory, or `--kv-cache-memory=70424346112` (65.59 GiB) to fully utilize gpu memory. Current kv cache memory in use is 58.36 GiB.
(EngineCore pid=25921) INFO 10-31 00:53:09 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=25921) INFO 10-31 00:53:09 [core.py:395] init engine (profile, create kv cache, warmup model) took 167.63 s (compilation: 24.79 s)
Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s]Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 3.68it/s]Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 7.14it/s]
(EngineCore pid=25921) INFO 10-31 00:53:09 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=25496) WARNING 10-31 00:53:10 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=25921) WARNING 10-31 00:53:10 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=25921) WARNING 10-31 00:53:11 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=25921) WARNING 10-31 00:53:12 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "00:54:09", "elapsed_min": 1.0, "submitted": 128, "finished": 18, "in_flight": 110, "gen_tokens": 57273, "tok_per_s": 955, "chain_tokens_median": 734.0, "chain_tokens_p90": 1303.8, "chain_tokens_max": 2064, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "00:55:09", "elapsed_min": 2.0, "submitted": 128, "finished": 31, "in_flight": 97, "gen_tokens": 160684, "tok_per_s": 1339, "chain_tokens_median": 866.5, "chain_tokens_p90": 2762.9, "chain_tokens_max": 4367, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
progress {"time": "00:56:09", "elapsed_min": 3.0, "submitted": 128, "finished": 61, "in_flight": 67, "gen_tokens": 408301, "tok_per_s": 2268, "chain_tokens_median": 969.0, "chain_tokens_p90": 3920.2000000000003, "chain_tokens_max": 6404, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
Xet Storage Details
- Size:
- 443 kB
- Xet hash:
- 85a12c2d044f64f94d08936ec4551e35781f13bfb3ec94f13bd66fdb4df0eab4
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.