Buckets:
| (MainProcess pid=153522) INFO 11-01 02:18:40 [model.py:733] Resolved architecture: LlamaForCausalLM | |
| (MainProcess pid=153522) INFO 11-01 02:18:40 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=153522) INFO 11-01 02:18:40 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048. | |
| (MainProcess pid=153522) INFO 11-01 02:18:40 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=153795) INFO 11-01 02:18:43 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/v2b_joint_merged', speculative_config=None, tokenizer='checkpoints/v2b_joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/v2b_joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=153795) INFO 11-01 02:18:44 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_024de2e71b2a4f748d89558f7b36c0e6 backend=nccl | |
| (EngineCore pid=153795) INFO 11-01 02:18:45 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=153795) INFO 11-01 02:18:45 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=153795) INFO 11-01 02:18:46 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=153795) INFO 11-01 02:18:47 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=153795) INFO 11-01 02:18:47 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=153795) INFO 11-01 02:18:47 [base_loader.py:82] Initializing model took 1.548803 seconds | |
| (EngineCore pid=153795) INFO 11-01 02:18:47 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 129.06 GiB. | |
| (EngineCore pid=153795) INFO 11-01 02:18:47 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.35it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.35it/s] | |
| (EngineCore pid=153795) INFO 11-01 02:18:48 [default_loader.py:501] Loading weights took 0.76 seconds | |
| (EngineCore pid=153795) INFO 11-01 02:18:50 [model_runner.py:424] Model loading took 4.75 GiB memory and 4.677984 seconds | |
| (EngineCore pid=153795) INFO 11-01 02:18:50 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=153795) INFO 11-01 02:18:50 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=153795) INFO 11-01 02:18:50 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=153795) INFO 11-01 02:18:51 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=153795) INFO 11-01 02:19:01 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/b830caab8c/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=153795) INFO 11-01 02:19:01 [backends.py:1166] [backbone] Dynamo bytecode transform time: 9.29 s | |
| (EngineCore pid=153795) INFO 11-01 02:19:08 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 7.49 s | |
| (EngineCore pid=153795) INFO 11-01 02:19:14 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4889803 bytes total | |
| (EngineCore pid=153795) INFO 11-01 02:19:14 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/8b11e22f345a5dc7763ac9fdd8eae668df4da66c816459ebdb605500ec426e7a/rank_0_0/model | |
| (EngineCore pid=153795) INFO 11-01 02:19:14 [monitor.py:62] [backbone] torch.compile took 22.52 s in total | |
| (EngineCore pid=153795) INFO 11-01 02:19:14 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.19 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:08, 6.04it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:00<00:04, 10.22it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:00<00:04, 11.17it/s] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:00<00:04, 9.90it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:00<00:03, 11.21it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:01<00:03, 10.63it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:01<00:03, 10.91it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:01<00:03, 11.85it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 17/51 [00:01<00:03, 9.73it/s] Capturing CUDA graphs (PIECEWISE): 37%|███▋ | 19/51 [00:01<00:02, 10.79it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 21/51 [00:01<00:02, 11.22it/s] Capturing CUDA graphs (PIECEWISE): 45%|████▌ | 23/51 [00:02<00:02, 9.90it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▉ | 25/51 [00:02<00:02, 10.66it/s] Capturing CUDA graphs (PIECEWISE): 53%|█████▎ | 27/51 [00:02<00:02, 9.80it/s] Capturing CUDA graphs (PIECEWISE): 57%|█████▋ | 29/51 [00:02<00:02, 10.69it/s] Capturing CUDA graphs (PIECEWISE): 61%|██████ | 31/51 [00:02<00:01, 11.14it/s] Capturing CUDA graphs (PIECEWISE): 65%|██████▍ | 33/51 [00:03<00:01, 10.76it/s] Capturing CUDA graphs (PIECEWISE): 69%|██████▊ | 35/51 [00:03<00:01, 11.63it/s] Capturing CUDA graphs (PIECEWISE): 73%|███████▎ | 37/51 [00:03<00:01, 11.93it/s] Capturing CUDA graphs (PIECEWISE): 76%|███████▋ | 39/51 [00:03<00:01, 10.62it/s] Capturing CUDA graphs (PIECEWISE): 80%|████████ | 41/51 [00:03<00:00, 11.45it/s] Capturing CUDA graphs (PIECEWISE): 84%|████████▍ | 43/51 [00:04<00:00, 10.31it/s] Capturing CUDA graphs (PIECEWISE): 88%|████████▊ | 45/51 [00:04<00:00, 11.12it/s] Capturing CUDA graphs (PIECEWISE): 92%|█████████▏| 47/51 [00:04<00:00, 11.83it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▌| 49/51 [00:04<00:00, 10.06it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:04<00:00, 10.52it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:04<00:00, 10.71it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 22.50it/s] | |
| (EngineCore pid=153795) INFO 11-01 02:19:22 [model_runner.py:1127] Graph capturing finished in 6 secs, took 0.40 GiB | |
| (EngineCore pid=153795) INFO 11-01 02:19:22 [gpu_worker.py:759] Available KV cache memory: 66.85 GiB | |
| (EngineCore pid=153795) INFO 11-01 02:19:22 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9143 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9257. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=153795) INFO 11-01 02:19:22 [kv_cache_utils.py:2327] GPU KV cache size: 1,668,928 tokens, Maximum concurrency for 32,768 tokens per request: 50.93x | |
| (EngineCore pid=153795) INFO 11-01 02:19:22 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=153795) INFO 11-01 02:19:22 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:03, 12.89it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:00<00:03, 12.60it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:00<00:04, 10.24it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:00<00:03, 11.34it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:00<00:04, 10.16it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:01<00:03, 11.18it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:01<00:03, 12.21it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:01<00:03, 10.52it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:01<00:03, 10.75it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:01<00:02, 11.29it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:01<00:02, 10.90it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:02<00:02, 11.67it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:02<00:02, 10.45it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:02<00:02, 11.15it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:02<00:01, 11.91it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:02<00:01, 10.19it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:03<00:01, 11.10it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:03<00:01, 11.92it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:03<00:01, 12.23it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:03<00:00, 12.63it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:03<00:00, 13.04it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:03<00:00, 13.05it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:03<00:00, 12.85it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:04<00:00, 13.15it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:04<00:00, 11.02it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:04<00:00, 11.51it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 24.27it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 23.19it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 19.64it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:01, 21.30it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 21.75it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 21.47it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 22.08it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 22.48it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 23.25it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 21.86it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 22.56it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 22.30it/s] | |
| (EngineCore pid=153795) INFO 11-01 02:19:47 [model_runner.py:1127] Graph capturing finished in 7 secs, took 0.23 GiB | |
| (EngineCore pid=153795) INFO 11-01 02:19:47 [gpu_worker.py:971] CUDA graph pool memory: 0.23 GiB (actual), 0.46 GiB (estimated), difference: 0.23 GiB (100.9%). | |
| (EngineCore pid=153795) INFO 11-01 02:19:47 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 5.1 GiB for consumed memory (weights + non-torch), 0.96 GiB for peak activation, and 0.23 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=71377237218` (66.48 GiB) to fit into requested memory, or `--kv-cache-memory=77741406720` (72.4 GiB) to fully utilize gpu memory. Current kv cache memory in use is 66.85 GiB. | |
| (EngineCore pid=153795) INFO 11-01 02:19:48 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=153795) INFO 11-01 02:19:48 [core.py:395] init engine (profile, create kv cache, warmup model) took 57.97 s (compilation: 22.52 s) | |
| (EngineCore pid=153795) INFO 11-01 02:19:48 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=153522) WARNING 11-01 02:19:50 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=153795) WARNING 11-01 02:19:50 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| readout v2b: 5957 items in 48s | |
| (EngineCore pid=153795) WARNING 11-01 02:20:37 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=153795) WARNING 11-01 02:20:38 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=153795) WARNING 11-01 02:20:39 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| progress {"time": "02:21:36", "elapsed_min": 1.0, "submitted": 420, "finished": 164, "in_flight": 256, "gen_tokens": 117898, "tok_per_s": 1965, "chain_tokens_median": 686.0, "chain_tokens_p90": 1076.6000000000004, "chain_tokens_max": 1319, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:21:36 [loggers.py:335] Engine 000: Avg prompt throughput: 20610.7 tokens/s, Avg generation throughput: 3204.8 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.6%, Prefix cache hit rate: 37.1% | |
| progress {"time": "02:22:36", "elapsed_min": 2.0, "submitted": 579, "finished": 323, "in_flight": 256, "gen_tokens": 286033, "tok_per_s": 2383, "chain_tokens_median": 740.0, "chain_tokens_p90": 1548.2, "chain_tokens_max": 2677, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:22:36 [loggers.py:335] Engine 000: Avg prompt throughput: 730.7 tokens/s, Avg generation throughput: 5817.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.2%, Prefix cache hit rate: 61.0% | |
| progress {"time": "02:23:36", "elapsed_min": 3.0, "submitted": 689, "finished": 433, "in_flight": 256, "gen_tokens": 434469, "tok_per_s": 2414, "chain_tokens_median": 762.0, "chain_tokens_p90": 2057.0000000000005, "chain_tokens_max": 3750, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:23:36 [loggers.py:335] Engine 000: Avg prompt throughput: 742.7 tokens/s, Avg generation throughput: 4608.8 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.8%, Prefix cache hit rate: 71.9% | |
| progress {"time": "02:24:36", "elapsed_min": 4.0, "submitted": 812, "finished": 556, "in_flight": 256, "gen_tokens": 621314, "tok_per_s": 2589, "chain_tokens_median": 776.5, "chain_tokens_p90": 2610.0, "chain_tokens_max": 4534, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:24:36 [loggers.py:335] Engine 000: Avg prompt throughput: 947.5 tokens/s, Avg generation throughput: 4377.7 tokens/s, Running: 254 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.9%, Prefix cache hit rate: 81.8% | |
| progress {"time": "02:25:36", "elapsed_min": 5.0, "submitted": 919, "finished": 663, "in_flight": 256, "gen_tokens": 813730, "tok_per_s": 2712, "chain_tokens_median": 802.0, "chain_tokens_p90": 2806.2000000000007, "chain_tokens_max": 5653, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:25:36 [loggers.py:335] Engine 000: Avg prompt throughput: 638.4 tokens/s, Avg generation throughput: 4067.4 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 48.2%, Prefix cache hit rate: 83.3% | |
| progress {"time": "02:26:36", "elapsed_min": 6.0, "submitted": 1031, "finished": 775, "in_flight": 256, "gen_tokens": 967494, "tok_per_s": 2687, "chain_tokens_median": 796.0, "chain_tokens_p90": 2893.2000000000007, "chain_tokens_max": 6657, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:26:36 [loggers.py:335] Engine 000: Avg prompt throughput: 480.9 tokens/s, Avg generation throughput: 3759.9 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.0%, Prefix cache hit rate: 83.9% | |
| progress {"time": "02:27:36", "elapsed_min": 7.0, "submitted": 1126, "finished": 870, "in_flight": 256, "gen_tokens": 1164015, "tok_per_s": 2771, "chain_tokens_median": 810.5, "chain_tokens_p90": 3232.1, "chain_tokens_max": 7374, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:27:36 [loggers.py:335] Engine 000: Avg prompt throughput: 310.0 tokens/s, Avg generation throughput: 3901.8 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.5%, Prefix cache hit rate: 86.2% | |
| progress {"time": "02:28:36", "elapsed_min": 8.0, "submitted": 1184, "finished": 928, "in_flight": 256, "gen_tokens": 1276993, "tok_per_s": 2660, "chain_tokens_median": 832.5, "chain_tokens_p90": 3288.5000000000005, "chain_tokens_max": 8336, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:28:36 [loggers.py:335] Engine 000: Avg prompt throughput: 217.4 tokens/s, Avg generation throughput: 4386.3 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 62.3%, Prefix cache hit rate: 87.1% | |
| progress {"time": "02:29:36", "elapsed_min": 9.0, "submitted": 1246, "finished": 990, "in_flight": 256, "gen_tokens": 1397915, "tok_per_s": 2589, "chain_tokens_median": 852.5, "chain_tokens_p90": 3357.500000000001, "chain_tokens_max": 8790, "parse_rate": 0.998989898989899, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:29:36 [loggers.py:335] Engine 000: Avg prompt throughput: 310.6 tokens/s, Avg generation throughput: 3797.3 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.8%, Prefix cache hit rate: 88.0% | |
| progress {"time": "02:30:36", "elapsed_min": 10.0, "submitted": 1306, "finished": 1050, "in_flight": 256, "gen_tokens": 1530805, "tok_per_s": 2551, "chain_tokens_median": 864.0, "chain_tokens_p90": 3453.5, "chain_tokens_max": 9535, "parse_rate": 0.9990476190476191, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:30:36 [loggers.py:335] Engine 000: Avg prompt throughput: 281.0 tokens/s, Avg generation throughput: 3507.4 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.3%, Prefix cache hit rate: 89.4% | |
| progress {"time": "02:31:36", "elapsed_min": 11.0, "submitted": 1362, "finished": 1106, "in_flight": 256, "gen_tokens": 1638603, "tok_per_s": 2483, "chain_tokens_median": 874.0, "chain_tokens_p90": 3577.0, "chain_tokens_max": 9535, "parse_rate": 0.9990958408679927, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:31:36 [loggers.py:335] Engine 000: Avg prompt throughput: 265.5 tokens/s, Avg generation throughput: 3383.8 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.0%, Prefix cache hit rate: 90.0% | |
| progress {"time": "02:32:36", "elapsed_min": 12.0, "submitted": 1409, "finished": 1153, "in_flight": 256, "gen_tokens": 1745814, "tok_per_s": 2425, "chain_tokens_median": 886.0, "chain_tokens_p90": 3725.599999999999, "chain_tokens_max": 9535, "parse_rate": 0.9991326973113617, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:32:36 [loggers.py:335] Engine 000: Avg prompt throughput: 217.3 tokens/s, Avg generation throughput: 3354.6 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 90.5% | |
| progress {"time": "02:33:36", "elapsed_min": 13.0, "submitted": 1455, "finished": 1199, "in_flight": 256, "gen_tokens": 1860784, "tok_per_s": 2386, "chain_tokens_median": 894.0, "chain_tokens_p90": 3836.2, "chain_tokens_max": 9535, "parse_rate": 0.9991659716430359, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:33:36 [loggers.py:335] Engine 000: Avg prompt throughput: 190.9 tokens/s, Avg generation throughput: 3218.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.4%, Prefix cache hit rate: 90.5% | |
| progress {"time": "02:34:36", "elapsed_min": 14.0, "submitted": 1493, "finished": 1237, "in_flight": 256, "gen_tokens": 1939041, "tok_per_s": 2308, "chain_tokens_median": 905.0, "chain_tokens_p90": 3837.4, "chain_tokens_max": 9535, "parse_rate": 0.9991915925626516, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:34:36 [loggers.py:335] Engine 000: Avg prompt throughput: 158.9 tokens/s, Avg generation throughput: 3274.1 tokens/s, Running: 256 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 90.7% | |
| progress {"time": "02:35:36", "elapsed_min": 15.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 2189, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:35:36 [loggers.py:335] Engine 000: Avg prompt throughput: 68.4 tokens/s, Avg generation throughput: 2857.3 tokens/s, Running: 211 reqs, Waiting: 45 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:36:36", "elapsed_min": 16.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 2052, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:36:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2514.8 tokens/s, Running: 159 reqs, Waiting: 97 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:37:36", "elapsed_min": 17.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1931, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:37:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2360.2 tokens/s, Running: 130 reqs, Waiting: 126 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:38:36", "elapsed_min": 18.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1824, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:38:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2223.1 tokens/s, Running: 111 reqs, Waiting: 145 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:39:36", "elapsed_min": 19.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1728, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:39:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1966.1 tokens/s, Running: 100 reqs, Waiting: 156 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:40:36", "elapsed_min": 20.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1642, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:40:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1777.2 tokens/s, Running: 91 reqs, Waiting: 165 reqs, GPU KV cache usage: 99.3%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:41:36", "elapsed_min": 21.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1564, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:41:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1611.5 tokens/s, Running: 85 reqs, Waiting: 171 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:42:36", "elapsed_min": 22.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1492, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:42:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1517.0 tokens/s, Running: 79 reqs, Waiting: 177 reqs, GPU KV cache usage: 99.0%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:43:36", "elapsed_min": 23.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1428, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:43:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1564.7 tokens/s, Running: 74 reqs, Waiting: 182 reqs, GPU KV cache usage: 99.0%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:44:36", "elapsed_min": 24.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1368, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:44:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1461.5 tokens/s, Running: 70 reqs, Waiting: 186 reqs, GPU KV cache usage: 99.3%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:45:36", "elapsed_min": 25.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1313, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:45:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1371.9 tokens/s, Running: 66 reqs, Waiting: 190 reqs, GPU KV cache usage: 99.0%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:46:36", "elapsed_min": 26.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1263, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:46:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1300.3 tokens/s, Running: 63 reqs, Waiting: 193 reqs, GPU KV cache usage: 99.4%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:47:36", "elapsed_min": 27.0, "submitted": 1511, "finished": 1255, "in_flight": 256, "gen_tokens": 1970161, "tok_per_s": 1216, "chain_tokens_median": 908.0, "chain_tokens_p90": 3830.800000000002, "chain_tokens_max": 9535, "parse_rate": 0.999203187250996, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:47:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1234.3 tokens/s, Running: 60 reqs, Waiting: 196 reqs, GPU KV cache usage: 99.4%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:48:36", "elapsed_min": 28.0, "submitted": 1512, "finished": 1256, "in_flight": 256, "gen_tokens": 1997627, "tok_per_s": 1189, "chain_tokens_median": 908.5, "chain_tokens_p90": 3836.5, "chain_tokens_max": 27466, "parse_rate": 0.9984076433121019, "looped": 1, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:48:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1077.8 tokens/s, Running: 58 reqs, Waiting: 198 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:49:36", "elapsed_min": 29.0, "submitted": 1514, "finished": 1258, "in_flight": 256, "gen_tokens": 2057009, "tok_per_s": 1182, "chain_tokens_median": 910.0, "chain_tokens_p90": 3845.199999999999, "chain_tokens_max": 29721, "parse_rate": 0.9968203497615262, "looped": 3, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:49:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1043.2 tokens/s, Running: 56 reqs, Waiting: 200 reqs, GPU KV cache usage: 98.3%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:50:36", "elapsed_min": 30.0, "submitted": 1515, "finished": 1259, "in_flight": 256, "gen_tokens": 2087154, "tok_per_s": 1159, "chain_tokens_median": 911.0, "chain_tokens_p90": 3862.6000000000004, "chain_tokens_max": 30145, "parse_rate": 0.9960285941223193, "looped": 4, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:50:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 985.9 tokens/s, Running: 54 reqs, Waiting: 202 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:51:36", "elapsed_min": 31.0, "submitted": 1520, "finished": 1264, "in_flight": 256, "gen_tokens": 2244331, "tok_per_s": 1207, "chain_tokens_median": 912.5, "chain_tokens_p90": 3932.5000000000027, "chain_tokens_max": 31589, "parse_rate": 0.9920886075949367, "looped": 9, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:51:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 677.9 tokens/s, Running: 54 reqs, Waiting: 202 reqs, GPU KV cache usage: 98.7%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:52:36", "elapsed_min": 32.0, "submitted": 1529, "finished": 1273, "in_flight": 256, "gen_tokens": 2529648, "tok_per_s": 1317, "chain_tokens_median": 917.0, "chain_tokens_p90": 4095.5999999999995, "chain_tokens_max": 31819, "parse_rate": 0.9850746268656716, "looped": 18, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:52:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 180.2 tokens/s, Running: 58 reqs, Waiting: 198 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:53:36", "elapsed_min": 33.0, "submitted": 1538, "finished": 1282, "in_flight": 256, "gen_tokens": 2817363, "tok_per_s": 1423, "chain_tokens_median": 919.5, "chain_tokens_p90": 4277.000000000003, "chain_tokens_max": 32104, "parse_rate": 0.9781591263650546, "looped": 27, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:53:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 275.1 tokens/s, Running: 66 reqs, Waiting: 190 reqs, GPU KV cache usage: 99.4%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:54:36", "elapsed_min": 34.0, "submitted": 1544, "finished": 1288, "in_flight": 256, "gen_tokens": 3009311, "tok_per_s": 1475, "chain_tokens_median": 924.5, "chain_tokens_p90": 4316.6, "chain_tokens_max": 32550, "parse_rate": 0.9736024844720497, "looped": 33, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:54:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 667.2 tokens/s, Running: 73 reqs, Waiting: 183 reqs, GPU KV cache usage: 99.4%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:55:36", "elapsed_min": 35.0, "submitted": 1549, "finished": 1293, "in_flight": 256, "gen_tokens": 3170975, "tok_per_s": 1510, "chain_tokens_median": 925.0, "chain_tokens_p90": 4362.4, "chain_tokens_max": 32579, "parse_rate": 0.9698375870069605, "looped": 38, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:55:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1042.4 tokens/s, Running: 75 reqs, Waiting: 181 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:56:36", "elapsed_min": 36.0, "submitted": 1558, "finished": 1302, "in_flight": 256, "gen_tokens": 3463067, "tok_per_s": 1603, "chain_tokens_median": 929.5, "chain_tokens_p90": 4480.6, "chain_tokens_max": 32635, "parse_rate": 0.9631336405529954, "looped": 47, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:56:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 333.0 tokens/s, Running: 140 reqs, Waiting: 116 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 90.9% | |
| progress {"time": "02:57:36", "elapsed_min": 37.0, "submitted": 1606, "finished": 1350, "in_flight": 256, "gen_tokens": 3705205, "tok_per_s": 1669, "chain_tokens_median": 970.0, "chain_tokens_p90": 4722.900000000007, "chain_tokens_max": 32635, "parse_rate": 0.9629629629629629, "looped": 48, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:57:36 [loggers.py:335] Engine 000: Avg prompt throughput: 4511.9 tokens/s, Avg generation throughput: 439.7 tokens/s, Running: 244 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 75.1% | |
| progress {"time": "02:58:36", "elapsed_min": 38.0, "submitted": 1634, "finished": 1378, "in_flight": 256, "gen_tokens": 3868105, "tok_per_s": 1696, "chain_tokens_median": 968.0, "chain_tokens_p90": 4794.9, "chain_tokens_max": 32635, "parse_rate": 0.9608127721335269, "looped": 52, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:58:36 [loggers.py:335] Engine 000: Avg prompt throughput: 750.0 tokens/s, Avg generation throughput: 2239.7 tokens/s, Running: 255 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 72.3% | |
| progress {"time": "02:59:36", "elapsed_min": 39.0, "submitted": 1686, "finished": 1430, "in_flight": 256, "gen_tokens": 4080801, "tok_per_s": 1744, "chain_tokens_median": 949.5, "chain_tokens_p90": 4794.3, "chain_tokens_max": 32658, "parse_rate": 0.9587412587412587, "looped": 57, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 02:59:36 [loggers.py:335] Engine 000: Avg prompt throughput: 684.8 tokens/s, Avg generation throughput: 2023.2 tokens/s, Running: 253 reqs, Waiting: 0 reqs, GPU KV cache usage: 94.4%, Prefix cache hit rate: 70.0% | |
| progress {"time": "03:00:36", "elapsed_min": 40.0, "submitted": 1730, "finished": 1474, "in_flight": 256, "gen_tokens": 4142149, "tok_per_s": 1726, "chain_tokens_median": 964.0, "chain_tokens_p90": 4701.900000000002, "chain_tokens_max": 32658, "parse_rate": 0.9599728629579376, "looped": 57, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:00:36 [loggers.py:335] Engine 000: Avg prompt throughput: 359.1 tokens/s, Avg generation throughput: 2121.7 tokens/s, Running: 254 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.5%, Prefix cache hit rate: 68.7% | |
| progress {"time": "03:01:36", "elapsed_min": 41.0, "submitted": 1768, "finished": 1512, "in_flight": 256, "gen_tokens": 4216273, "tok_per_s": 1714, "chain_tokens_median": 954.0, "chain_tokens_p90": 4651.5, "chain_tokens_max": 32658, "parse_rate": 0.9603174603174603, "looped": 58, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:01:36 [loggers.py:335] Engine 000: Avg prompt throughput: 288.6 tokens/s, Avg generation throughput: 2120.4 tokens/s, Running: 235 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:02:36", "elapsed_min": 42.0, "submitted": 1768, "finished": 1512, "in_flight": 256, "gen_tokens": 4216273, "tok_per_s": 1673, "chain_tokens_median": 954.0, "chain_tokens_p90": 4651.5, "chain_tokens_max": 32658, "parse_rate": 0.9603174603174603, "looped": 58, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:02:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2283.0 tokens/s, Running: 159 reqs, Waiting: 97 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:03:36", "elapsed_min": 43.0, "submitted": 1768, "finished": 1512, "in_flight": 256, "gen_tokens": 4216273, "tok_per_s": 1634, "chain_tokens_median": 954.0, "chain_tokens_p90": 4651.5, "chain_tokens_max": 32658, "parse_rate": 0.9603174603174603, "looped": 58, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:03:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1646.3 tokens/s, Running: 134 reqs, Waiting: 122 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:04:36", "elapsed_min": 44.0, "submitted": 1770, "finished": 1514, "in_flight": 256, "gen_tokens": 4280996, "tok_per_s": 1621, "chain_tokens_median": 956.0, "chain_tokens_p90": 4668.4, "chain_tokens_max": 32658, "parse_rate": 0.9590488771466315, "looped": 60, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:04:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1441.4 tokens/s, Running: 127 reqs, Waiting: 128 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:05:36", "elapsed_min": 45.0, "submitted": 1771, "finished": 1515, "in_flight": 256, "gen_tokens": 4313358, "tok_per_s": 1597, "chain_tokens_median": 956.0, "chain_tokens_p90": 4697.200000000006, "chain_tokens_max": 32658, "parse_rate": 0.9584158415841584, "looped": 61, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:05:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1519.2 tokens/s, Running: 109 reqs, Waiting: 147 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:06:36", "elapsed_min": 46.0, "submitted": 1771, "finished": 1515, "in_flight": 256, "gen_tokens": 4313358, "tok_per_s": 1563, "chain_tokens_median": 956.0, "chain_tokens_p90": 4697.200000000006, "chain_tokens_max": 32658, "parse_rate": 0.9584158415841584, "looped": 61, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:06:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1377.4 tokens/s, Running: 100 reqs, Waiting: 156 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:07:36", "elapsed_min": 47.0, "submitted": 1774, "finished": 1518, "in_flight": 256, "gen_tokens": 4410249, "tok_per_s": 1564, "chain_tokens_median": 959.0, "chain_tokens_p90": 4775.0999999999985, "chain_tokens_max": 32658, "parse_rate": 0.9565217391304348, "looped": 64, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:07:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1109.0 tokens/s, Running: 101 reqs, Waiting: 155 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:08:36", "elapsed_min": 48.0, "submitted": 1775, "finished": 1519, "in_flight": 256, "gen_tokens": 4442844, "tok_per_s": 1543, "chain_tokens_median": 960.0, "chain_tokens_p90": 4794.6, "chain_tokens_max": 32658, "parse_rate": 0.955892034233048, "looped": 65, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:08:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1312.2 tokens/s, Running: 94 reqs, Waiting: 162 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:09:36", "elapsed_min": 49.0, "submitted": 1777, "finished": 1521, "in_flight": 256, "gen_tokens": 4507652, "tok_per_s": 1533, "chain_tokens_median": 964.0, "chain_tokens_p90": 4804.0, "chain_tokens_max": 32658, "parse_rate": 0.9546351084812623, "looped": 67, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:09:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1156.9 tokens/s, Running: 92 reqs, Waiting: 164 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:10:36", "elapsed_min": 50.0, "submitted": 1779, "finished": 1523, "in_flight": 256, "gen_tokens": 4572309, "tok_per_s": 1524, "chain_tokens_median": 964.0, "chain_tokens_p90": 4822.2, "chain_tokens_max": 32658, "parse_rate": 0.953381483913329, "looped": 69, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:10:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1112.6 tokens/s, Running: 89 reqs, Waiting: 167 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:11:36", "elapsed_min": 51.0, "submitted": 1780, "finished": 1524, "in_flight": 256, "gen_tokens": 4603861, "tok_per_s": 1504, "chain_tokens_median": 965.5, "chain_tokens_p90": 4831.700000000001, "chain_tokens_max": 32658, "parse_rate": 0.952755905511811, "looped": 70, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:11:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1226.9 tokens/s, Running: 84 reqs, Waiting: 172 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:12:36", "elapsed_min": 52.0, "submitted": 1782, "finished": 1526, "in_flight": 256, "gen_tokens": 4668171, "tok_per_s": 1496, "chain_tokens_median": 968.0, "chain_tokens_p90": 4907.5, "chain_tokens_max": 32658, "parse_rate": 0.9515072083879423, "looped": 72, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:12:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1087.4 tokens/s, Running: 82 reqs, Waiting: 174 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:13:36", "elapsed_min": 53.0, "submitted": 1784, "finished": 1528, "in_flight": 256, "gen_tokens": 4732298, "tok_per_s": 1488, "chain_tokens_median": 970.0, "chain_tokens_p90": 4914.5, "chain_tokens_max": 32658, "parse_rate": 0.9502617801047121, "looped": 74, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:13:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1263.2 tokens/s, Running: 79 reqs, Waiting: 177 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:14:36", "elapsed_min": 54.0, "submitted": 1788, "finished": 1532, "in_flight": 256, "gen_tokens": 4859630, "tok_per_s": 1500, "chain_tokens_median": 973.0, "chain_tokens_p90": 4960.400000000001, "chain_tokens_max": 32658, "parse_rate": 0.9477806788511749, "looped": 78, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:14:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 950.0 tokens/s, Running: 83 reqs, Waiting: 173 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:15:36", "elapsed_min": 55.0, "submitted": 1790, "finished": 1534, "in_flight": 256, "gen_tokens": 4923514, "tok_per_s": 1492, "chain_tokens_median": 976.0, "chain_tokens_p90": 5007.600000000002, "chain_tokens_max": 32658, "parse_rate": 0.9465449804432855, "looped": 80, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:15:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1265.7 tokens/s, Running: 80 reqs, Waiting: 176 reqs, GPU KV cache usage: 99.3%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:16:36", "elapsed_min": 56.0, "submitted": 1792, "finished": 1536, "in_flight": 256, "gen_tokens": 4988128, "tok_per_s": 1484, "chain_tokens_median": 978.0, "chain_tokens_p90": 5035.0, "chain_tokens_max": 32658, "parse_rate": 0.9453125, "looped": 82, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:16:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1276.7 tokens/s, Running: 77 reqs, Waiting: 179 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:17:36", "elapsed_min": 57.0, "submitted": 1795, "finished": 1539, "in_flight": 256, "gen_tokens": 5084311, "tok_per_s": 1487, "chain_tokens_median": 979.0, "chain_tokens_p90": 5107.6, "chain_tokens_max": 32658, "parse_rate": 0.9434697855750487, "looped": 85, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:17:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1150.7 tokens/s, Running: 78 reqs, Waiting: 178 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:18:36", "elapsed_min": 58.0, "submitted": 1798, "finished": 1542, "in_flight": 256, "gen_tokens": 5179843, "tok_per_s": 1488, "chain_tokens_median": 979.5, "chain_tokens_p90": 5145.6, "chain_tokens_max": 32658, "parse_rate": 0.9416342412451362, "looped": 88, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:18:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1076.2 tokens/s, Running: 80 reqs, Waiting: 176 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:19:36", "elapsed_min": 59.0, "submitted": 1801, "finished": 1545, "in_flight": 256, "gen_tokens": 5276087, "tok_per_s": 1490, "chain_tokens_median": 982.0, "chain_tokens_p90": 5187.800000000007, "chain_tokens_max": 32658, "parse_rate": 0.9398058252427185, "looped": 91, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:19:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1053.2 tokens/s, Running: 86 reqs, Waiting: 170 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:20:36", "elapsed_min": 60.0, "submitted": 1802, "finished": 1546, "in_flight": 256, "gen_tokens": 5308643, "tok_per_s": 1475, "chain_tokens_median": 983.0, "chain_tokens_p90": 5212.5, "chain_tokens_max": 32658, "parse_rate": 0.9391979301423027, "looped": 92, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:20:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1334.1 tokens/s, Running: 75 reqs, Waiting: 181 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:21:36", "elapsed_min": 61.0, "submitted": 1806, "finished": 1550, "in_flight": 256, "gen_tokens": 5437217, "tok_per_s": 1485, "chain_tokens_median": 984.5, "chain_tokens_p90": 5286.500000000002, "chain_tokens_max": 32658, "parse_rate": 0.9367741935483871, "looped": 96, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:21:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 943.9 tokens/s, Running: 92 reqs, Waiting: 164 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:22:36", "elapsed_min": 62.0, "submitted": 1808, "finished": 1552, "in_flight": 256, "gen_tokens": 5501727, "tok_per_s": 1479, "chain_tokens_median": 985.0, "chain_tokens_p90": 5399.00000000001, "chain_tokens_max": 32658, "parse_rate": 0.9355670103092784, "looped": 98, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:22:36 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1075.8 tokens/s, Running: 94 reqs, Waiting: 162 reqs, GPU KV cache usage: 99.9%, Prefix cache hit rate: 68.0% | |
| progress {"time": "03:23:36", "elapsed_min": 63.0, "submitted": 1862, "finished": 1606, "in_flight": 256, "gen_tokens": 5706358, "tok_per_s": 1510, "chain_tokens_median": 1010.0, "chain_tokens_p90": 5355.0, "chain_tokens_max": 32658, "parse_rate": 0.935865504358655, "looped": 101, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=153522) INFO 11-01 03:23:36 [loggers.py:335] Engine 000: Avg prompt throughput: 2210.4 tokens/s, Avg generation throughput: 671.1 tokens/s, Running: 133 reqs, Waiting: 122 reqs, GPU KV cache usage: 99.3%, Prefix cache hit rate: 60.9% | |
| (MainProcess pid=156107) INFO 11-01 03:24:39 [model.py:733] Resolved architecture: LlamaForCausalLM | |
| (MainProcess pid=156107) INFO 11-01 03:24:39 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=156107) INFO 11-01 03:24:39 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048. | |
| (MainProcess pid=156107) INFO 11-01 03:24:39 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=156379) INFO 11-01 03:24:41 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/v2b_joint_merged', speculative_config=None, tokenizer='checkpoints/v2b_joint_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/v2b_joint_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 192, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=156379) INFO 11-01 03:24:43 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_5ff3fc7a187d4427a2292bb32bac974b backend=nccl | |
| (EngineCore pid=156379) INFO 11-01 03:24:43 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=156379) INFO 11-01 03:24:43 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=156379) INFO 11-01 03:24:44 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=156379) INFO 11-01 03:24:45 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=156379) INFO 11-01 03:24:45 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=156379) INFO 11-01 03:24:45 [base_loader.py:82] Initializing model took 1.454567 seconds | |
| (EngineCore pid=156379) INFO 11-01 03:24:45 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 129.07 GiB. | |
| (EngineCore pid=156379) INFO 11-01 03:24:45 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.43it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.43it/s] | |
| (EngineCore pid=156379) INFO 11-01 03:24:46 [default_loader.py:501] Loading weights took 0.72 seconds | |
| (EngineCore pid=156379) INFO 11-01 03:24:48 [model_runner.py:424] Model loading took 4.75 GiB memory and 3.939792 seconds | |
| (EngineCore pid=156379) INFO 11-01 03:24:48 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=156379) INFO 11-01 03:24:48 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=156379) INFO 11-01 03:24:48 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=156379) INFO 11-01 03:24:49 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=156379) INFO 11-01 03:24:58 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/96dd63e7a0/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=156379) INFO 11-01 03:24:58 [backends.py:1166] [backbone] Dynamo bytecode transform time: 9.24 s | |
| (EngineCore pid=156379) INFO 11-01 03:25:07 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 8.80 s | |
| (EngineCore pid=156379) INFO 11-01 03:25:12 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4889380 bytes total | |
| (EngineCore pid=156379) INFO 11-01 03:25:13 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/aa3f47b3a73c9a7c129db57924b313997a3010322f7d686911c02255ebf37710/rank_0_0/model | |
| (EngineCore pid=156379) INFO 11-01 03:25:13 [monitor.py:62] [backbone] torch.compile took 23.91 s in total | |
| (EngineCore pid=156379) INFO 11-01 03:25:13 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.22 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/27 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 7%|▋ | 2/27 [00:00<00:01, 17.03it/s] Capturing CUDA graphs (PIECEWISE): 19%|█▊ | 5/27 [00:00<00:01, 19.96it/s] Capturing CUDA graphs (PIECEWISE): 30%|██▉ | 8/27 [00:00<00:00, 21.30it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 11/27 [00:00<00:00, 21.92it/s] Capturing CUDA graphs (PIECEWISE): 52%|█████▏ | 14/27 [00:00<00:00, 21.31it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 17/27 [00:00<00:00, 21.08it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 20/27 [00:00<00:00, 20.81it/s] Capturing CUDA graphs (PIECEWISE): 85%|████████▌ | 23/27 [00:01<00:00, 20.69it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▋| 26/27 [00:01<00:00, 19.45it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:01<00:00, 20.04it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 25.39it/s] | |
| (EngineCore pid=156379) INFO 11-01 03:25:17 [model_runner.py:1127] Graph capturing finished in 2 secs, took 0.24 GiB | |
| (EngineCore pid=156379) INFO 11-01 03:25:18 [gpu_worker.py:759] Available KV cache memory: 67.09 GiB | |
| (EngineCore pid=156379) INFO 11-01 03:25:18 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9164 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9236. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=156379) INFO 11-01 03:25:18 [kv_cache_utils.py:2327] GPU KV cache size: 1,675,072 tokens, Maximum concurrency for 32,768 tokens per request: 51.12x | |
| (EngineCore pid=156379) INFO 11-01 03:25:18 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=156379) INFO 11-01 03:25:18 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/27 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 11%|█ | 3/27 [00:00<00:01, 21.47it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 6/27 [00:00<00:00, 22.55it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 9/27 [00:00<00:00, 22.91it/s] Capturing CUDA graphs (PIECEWISE): 44%|████▍ | 12/27 [00:00<00:00, 23.14it/s] Capturing CUDA graphs (PIECEWISE): 56%|█████▌ | 15/27 [00:00<00:00, 22.57it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 18/27 [00:00<00:00, 22.25it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 21/27 [00:00<00:00, 21.95it/s] Capturing CUDA graphs (PIECEWISE): 89%|████████▉ | 24/27 [00:01<00:00, 21.57it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:01<00:00, 20.76it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:01<00:00, 21.72it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/15 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 20%|██ | 3/15 [00:00<00:00, 24.65it/s] Capturing CUDA graphs (FULL): 40%|████ | 6/15 [00:00<00:00, 24.72it/s] Capturing CUDA graphs (FULL): 60%|██████ | 9/15 [00:00<00:00, 24.41it/s] Capturing CUDA graphs (FULL): 80%|████████ | 12/15 [00:00<00:00, 24.91it/s] Capturing CUDA graphs (FULL): 100%|██████████| 15/15 [00:00<00:00, 25.29it/s] Capturing CUDA graphs (FULL): 100%|██████████| 15/15 [00:00<00:00, 25.01it/s] | |
| (EngineCore pid=156379) INFO 11-01 03:25:38 [model_runner.py:1127] Graph capturing finished in 3 secs, took 0.12 GiB | |
| (EngineCore pid=156379) INFO 11-01 03:25:38 [gpu_worker.py:971] CUDA graph pool memory: 0.12 GiB (actual), 0.29 GiB (estimated), difference: 0.17 GiB (146.7%). | |
| (EngineCore pid=156379) INFO 11-01 03:25:38 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 5.02 GiB for consumed memory (weights + non-torch), 0.79 GiB for peak activation, and 0.12 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=71758921954` (66.83 GiB) to fit into requested memory, or `--kv-cache-memory=78123091456` (72.76 GiB) to fully utilize gpu memory. Current kv cache memory in use is 67.09 GiB. | |
| (EngineCore pid=156379) INFO 11-01 03:25:38 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=156379) INFO 11-01 03:25:39 [core.py:395] init engine (profile, create kv cache, warmup model) took 51.20 s (compilation: 23.91 s) | |
| (EngineCore pid=156379) INFO 11-01 03:25:39 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| readout v2b: exists | |
| (MainProcess pid=156107) WARNING 11-01 03:25:40 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=156379) WARNING 11-01 03:25:41 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=156379) WARNING 11-01 03:25:41 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=156379) WARNING 11-01 03:25:42 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| progress {"time": "03:26:39", "elapsed_min": 1.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:26:39 [loggers.py:335] Engine 000: Avg prompt throughput: 1432.9 tokens/s, Avg generation throughput: 4778.9 tokens/s, Running: 96 reqs, Waiting: 159 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 7.4% | |
| progress {"time": "03:27:39", "elapsed_min": 2.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:27:39 [loggers.py:335] Engine 000: Avg prompt throughput: 597.1 tokens/s, Avg generation throughput: 4105.7 tokens/s, Running: 96 reqs, Waiting: 160 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 7.5% | |
| (EngineCore pid=156379) WARNING 11-01 03:28:00 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| progress {"time": "03:28:39", "elapsed_min": 3.0, "submitted": 391, "finished": 162, "in_flight": 229, "gen_tokens": 313542, "tok_per_s": 1742, "chain_tokens_median": 1244.5, "chain_tokens_p90": 4499.6, "chain_tokens_max": 6212, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:28:39 [loggers.py:335] Engine 000: Avg prompt throughput: 509.6 tokens/s, Avg generation throughput: 3066.4 tokens/s, Running: 96 reqs, Waiting: 133 reqs, GPU KV cache usage: 25.2%, Prefix cache hit rate: 72.2% | |
| progress {"time": "03:29:39", "elapsed_min": 4.0, "submitted": 391, "finished": 164, "in_flight": 227, "gen_tokens": 314699, "tok_per_s": 1311, "chain_tokens_median": 1233.5, "chain_tokens_p90": 4494.8, "chain_tokens_max": 6212, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:29:39 [loggers.py:335] Engine 000: Avg prompt throughput: 206.0 tokens/s, Avg generation throughput: 3050.2 tokens/s, Running: 96 reqs, Waiting: 131 reqs, GPU KV cache usage: 29.6%, Prefix cache hit rate: 70.8% | |
| progress {"time": "03:30:39", "elapsed_min": 5.0, "submitted": 391, "finished": 164, "in_flight": 227, "gen_tokens": 314699, "tok_per_s": 1049, "chain_tokens_median": 1233.5, "chain_tokens_p90": 4494.8, "chain_tokens_max": 6212, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:30:39 [loggers.py:335] Engine 000: Avg prompt throughput: 177.7 tokens/s, Avg generation throughput: 2770.6 tokens/s, Running: 96 reqs, Waiting: 131 reqs, GPU KV cache usage: 36.9%, Prefix cache hit rate: 69.5% | |
| progress {"time": "03:31:39", "elapsed_min": 6.0, "submitted": 391, "finished": 298, "in_flight": 93, "gen_tokens": 552070, "tok_per_s": 1533, "chain_tokens_median": 969.5, "chain_tokens_p90": 4459.0, "chain_tokens_max": 12053, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:31:39 [loggers.py:335] Engine 000: Avg prompt throughput: 84.8 tokens/s, Avg generation throughput: 2085.0 tokens/s, Running: 93 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.8%, Prefix cache hit rate: 79.4% | |
| progress {"time": "03:32:39", "elapsed_min": 7.0, "submitted": 391, "finished": 318, "in_flight": 73, "gen_tokens": 627336, "tok_per_s": 1494, "chain_tokens_median": 1053.5, "chain_tokens_p90": 4782.0, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:32:39 [loggers.py:335] Engine 000: Avg prompt throughput: 3.1 tokens/s, Avg generation throughput: 2675.8 tokens/s, Running: 73 reqs, Waiting: 0 reqs, GPU KV cache usage: 46.4%, Prefix cache hit rate: 81.2% | |
| progress {"time": "03:33:39", "elapsed_min": 8.0, "submitted": 391, "finished": 323, "in_flight": 68, "gen_tokens": 658558, "tok_per_s": 1372, "chain_tokens_median": 1073.0, "chain_tokens_p90": 4898.8, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:33:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.8 tokens/s, Avg generation throughput: 2495.2 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.4%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:34:39", "elapsed_min": 9.0, "submitted": 391, "finished": 323, "in_flight": 68, "gen_tokens": 658558, "tok_per_s": 1220, "chain_tokens_median": 1073.0, "chain_tokens_p90": 4898.8, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:34:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2178.2 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 61.2%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:35:39", "elapsed_min": 10.0, "submitted": 391, "finished": 323, "in_flight": 68, "gen_tokens": 658558, "tok_per_s": 1098, "chain_tokens_median": 1073.0, "chain_tokens_p90": 4898.8, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:35:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1977.6 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.2%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:36:39", "elapsed_min": 11.0, "submitted": 391, "finished": 323, "in_flight": 68, "gen_tokens": 658558, "tok_per_s": 998, "chain_tokens_median": 1073.0, "chain_tokens_p90": 4898.8, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:36:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1823.5 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.8%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:37:39", "elapsed_min": 12.0, "submitted": 391, "finished": 323, "in_flight": 68, "gen_tokens": 658558, "tok_per_s": 915, "chain_tokens_median": 1073.0, "chain_tokens_p90": 4898.8, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:37:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1701.1 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.9%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:38:39", "elapsed_min": 13.0, "submitted": 391, "finished": 323, "in_flight": 68, "gen_tokens": 658558, "tok_per_s": 844, "chain_tokens_median": 1073.0, "chain_tokens_p90": 4898.8, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:38:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1601.3 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.6%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:39:39", "elapsed_min": 14.0, "submitted": 391, "finished": 323, "in_flight": 68, "gen_tokens": 658558, "tok_per_s": 784, "chain_tokens_median": 1073.0, "chain_tokens_p90": 4898.8, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:39:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1516.4 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 92.0%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:40:39", "elapsed_min": 15.0, "submitted": 391, "finished": 323, "in_flight": 68, "gen_tokens": 658558, "tok_per_s": 732, "chain_tokens_median": 1073.0, "chain_tokens_p90": 4898.8, "chain_tokens_max": 12832, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:40:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1443.8 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.2%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:41:39", "elapsed_min": 16.0, "submitted": 391, "finished": 324, "in_flight": 67, "gen_tokens": 683401, "tok_per_s": 712, "chain_tokens_median": 1078.5, "chain_tokens_p90": 4991.299999999998, "chain_tokens_max": 24843, "parse_rate": 0.9969135802469136, "looped": 1, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:41:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1383.8 tokens/s, Running: 66 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:42:39", "elapsed_min": 17.0, "submitted": 391, "finished": 325, "in_flight": 66, "gen_tokens": 708376, "tok_per_s": 694, "chain_tokens_median": 1084.0, "chain_tokens_p90": 5057.000000000001, "chain_tokens_max": 24975, "parse_rate": 0.9938461538461538, "looped": 2, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:42:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1215.9 tokens/s, Running: 63 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:43:39", "elapsed_min": 18.0, "submitted": 391, "finished": 325, "in_flight": 66, "gen_tokens": 708376, "tok_per_s": 656, "chain_tokens_median": 1084.0, "chain_tokens_p90": 5057.000000000001, "chain_tokens_max": 24975, "parse_rate": 0.9938461538461538, "looped": 2, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:43:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1246.0 tokens/s, Running: 59 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.0%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:44:39", "elapsed_min": 19.0, "submitted": 391, "finished": 331, "in_flight": 60, "gen_tokens": 896960, "tok_per_s": 787, "chain_tokens_median": 1103.0, "chain_tokens_p90": 5533.0, "chain_tokens_max": 31672, "parse_rate": 0.9758308157099698, "looped": 8, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:44:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 738.0 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.0%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:45:39", "elapsed_min": 20.0, "submitted": 391, "finished": 348, "in_flight": 43, "gen_tokens": 1443453, "tok_per_s": 1203, "chain_tokens_median": 1220.5, "chain_tokens_p90": 6386.600000000001, "chain_tokens_max": 32609, "parse_rate": 0.9281609195402298, "looped": 25, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:45:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1182.2 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.0%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:46:39", "elapsed_min": 21.0, "submitted": 391, "finished": 351, "in_flight": 40, "gen_tokens": 1538679, "tok_per_s": 1221, "chain_tokens_median": 1240.0, "chain_tokens_p90": 6943.0, "chain_tokens_max": 32609, "parse_rate": 0.9202279202279202, "looped": 28, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:46:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1146.6 tokens/s, Running: 40 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.2%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:47:39", "elapsed_min": 22.0, "submitted": 391, "finished": 360, "in_flight": 31, "gen_tokens": 1829804, "tok_per_s": 1386, "chain_tokens_median": 1271.5, "chain_tokens_p90": 24856.200000000004, "chain_tokens_max": 32653, "parse_rate": 0.8972222222222223, "looped": 37, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:47:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1088.9 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.6%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:48:39", "elapsed_min": 23.0, "submitted": 391, "finished": 369, "in_flight": 22, "gen_tokens": 2121216, "tok_per_s": 1537, "chain_tokens_median": 1378.0, "chain_tokens_p90": 31490.2, "chain_tokens_max": 32653, "parse_rate": 0.8753387533875339, "looped": 46, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:48:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 981.7 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.5%, Prefix cache hit rate: 81.8% | |
| progress {"time": "03:49:39", "elapsed_min": 24.0, "submitted": 391, "finished": 386, "in_flight": 5, "gen_tokens": 2672795, "tok_per_s": 1856, "chain_tokens_median": 1582.5, "chain_tokens_p90": 32301.0, "chain_tokens_max": 32654, "parse_rate": 0.8367875647668394, "looped": 63, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=156107) INFO 11-01 03:49:39 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 711.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 81.8% | |
| reason v2b: {"submitted": 391, "finished": 391, "gen_tokens": 2835202, "t0": 1793503539.7389824, "seconds": 1485.1973476409912, "tok_per_s": 1908.9732448709829} | |
| reason v2b: post-chain readout 1844 taken at generation, 0 by prefill | |
| reason v2b: 2013 items, post-readout in 1486s total | |
| latency v2b: {"readout_ms": {"median": 21.76104485988617, "p90": 30.89566342532635, "n": 100}, "reasoning_ms": {"median": 6353.157198056579, "p90": 209582.13100507855, "n": 20}} | |
| (MainProcess pid=156107) INFO 11-01 04:03:07 [core_client.py:770] [shutdown] MPClient: start timeout=default | |
| (MainProcess pid=156107) INFO 11-01 04:03:07 [core_client.py:772] [shutdown] MPClient: stopping engine manager | |
| (MainProcess pid=156107) INFO 11-01 04:03:07 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=156379) INFO 11-01 04:03:07 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=156379) INFO 11-01 04:03:07 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=156379) INFO 11-01 04:03:07 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=156379) INFO 11-01 04:03:07 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=156107) INFO 11-01 04:03:10 [core_client.py:774] [shutdown] MPClient: engine manager stopped | |
| (MainProcess pid=156107) INFO 11-01 04:03:10 [core_client.py:775] [shutdown] MPClient: cleaning up background resources | |
| (MainProcess pid=156107) INFO 11-01 04:03:10 [core_client.py:777] [shutdown] MPClient: complete | |
Xet Storage Details
- Size:
- 83 kB
- Xet hash:
- 6a7a17ffd666b9788a3a8d6cc3c74bb4723aeb19c1e1b5d75a8abd644bdd09e4
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.