Buckets:

Pranav2748's picture
download
raw
40.9 kB
(MainProcess pid=237959) INFO 11-01 06:16:02 [model.py:733] Resolved architecture: Qwen3_5ForCausalLM
(MainProcess pid=237959) INFO 11-01 06:16:02 [model.py:2152] Using max model len 32768
(MainProcess pid=237959) INFO 11-01 06:16:02 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048.
(MainProcess pid=237959) INFO 11-01 06:16:02 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForCausalLM by default when prefix caching is enabled
(MainProcess pid=237959) INFO 11-01 06:16:02 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=238247) INFO 11-01 06:16:08 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/rl40_merged', speculative_config=None, tokenizer='checkpoints/rl40_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/rl40_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 320, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=238247) INFO 11-01 06:16:09 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_105391ea65aa460882b5d448181936b4 backend=nccl
(EngineCore pid=238247) INFO 11-01 06:16:09 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=238247) INFO 11-01 06:16:09 [gpu_worker.py:518] Using V2 Model Runner
(EngineCore pid=238247) INFO 11-01 06:16:11 [model_runner.py:392] Loading model from scratch...
(EngineCore pid=238247) INFO 11-01 06:16:11 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=238247) INFO 11-01 06:16:12 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=238247) INFO 11-01 06:16:12 [flash_attn.py:1117] Using FlashAttention version 2
(EngineCore pid=238247) INFO 11-01 06:16:12 [base_loader.py:82] Initializing model took 1.473896 seconds
(EngineCore pid=238247) INFO 11-01 06:16:12 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 7.83 GiB. Available RAM: 114.12 GiB.
(EngineCore pid=238247) INFO 11-01 06:16:12 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.10s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.10s/it]
(EngineCore pid=238247) INFO 11-01 06:16:13 [default_loader.py:501] Loading weights took 1.26 seconds
(EngineCore pid=238247) INFO 11-01 06:16:15 [model_runner.py:424] Model loading took 7.94 GiB memory and 4.503753 seconds
(EngineCore pid=238247) INFO 11-01 06:16:15 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=238247) INFO 11-01 06:16:15 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=238247) INFO 11-01 06:16:15 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=238247) INFO 11-01 06:16:15 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=238247) INFO 11-01 06:16:15 [utils.py:347] Using LBNHC KV cache layout.
(EngineCore pid=238247) INFO 11-01 06:16:16 [monitor.py:40] [backbone] Starting torch.compile
(EngineCore pid=238247) INFO 11-01 06:16:21 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/02d80b4254/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=238247) INFO 11-01 06:16:21 [backends.py:1166] [backbone] Dynamo bytecode transform time: 5.23 s
(EngineCore pid=238247) INFO 11-01 06:16:40 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 18.36 s
(EngineCore pid=238247) INFO 11-01 06:16:42 [backends.py:927] collected artifacts: 33 entries, 12 artifacts, 20433542 bytes total
(EngineCore pid=238247) INFO 11-01 06:16:42 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/21dc3c05a01f74fb45291f91b82fd96ae99118348a18fb6acd1765fa799b0a38/rank_0_0/model
(EngineCore pid=238247) INFO 11-01 06:16:42 [monitor.py:62] [backbone] torch.compile took 26.61 s in total
(EngineCore pid=238247) INFO 11-01 06:18:11 [monitor.py:91] [backbone] Initial profiling/warmup run took 88.10 s
Capturing CUDA graphs (PIECEWISE): 0%| | 0/39 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/39 [00:00<00:32, 1.18it/s] Capturing CUDA graphs (PIECEWISE): 5%|▌ | 2/39 [00:01<00:27, 1.35it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 3/39 [00:01<00:19, 1.84it/s] Capturing CUDA graphs (PIECEWISE): 10%|█ | 4/39 [00:02<00:15, 2.24it/s] Capturing CUDA graphs (PIECEWISE): 13%|█▎ | 5/39 [00:02<00:13, 2.59it/s] Capturing CUDA graphs (PIECEWISE): 15%|█▌ | 6/39 [00:02<00:10, 3.01it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 7/39 [00:02<00:09, 3.39it/s] Capturing CUDA graphs (PIECEWISE): 21%|██ | 8/39 [00:03<00:08, 3.67it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 9/39 [00:03<00:08, 3.70it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 10/39 [00:03<00:08, 3.52it/s] Capturing CUDA graphs (PIECEWISE): 28%|██▊ | 11/39 [00:03<00:07, 3.61it/s] Capturing CUDA graphs (PIECEWISE): 31%|███ | 12/39 [00:04<00:07, 3.76it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 13/39 [00:04<00:06, 3.92it/s] Capturing CUDA graphs (PIECEWISE): 36%|███▌ | 14/39 [00:04<00:06, 4.14it/s] Capturing CUDA graphs (PIECEWISE): 38%|███▊ | 15/39 [00:04<00:05, 4.77it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 16/39 [00:04<00:04, 5.52it/s] Capturing CUDA graphs (PIECEWISE): 44%|████▎ | 17/39 [00:05<00:09, 2.30it/s] Capturing CUDA graphs (PIECEWISE): 46%|████▌ | 18/39 [00:06<00:07, 2.73it/s] Capturing CUDA graphs (PIECEWISE): 54%|█████▍ | 21/39 [00:06<00:03, 5.31it/s] Capturing CUDA graphs (PIECEWISE): 62%|██████▏ | 24/39 [00:06<00:01, 7.95it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 26/39 [00:06<00:01, 9.69it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 29/39 [00:06<00:00, 12.29it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 32/39 [00:06<00:00, 14.50it/s] Capturing CUDA graphs (PIECEWISE): 90%|████████▉ | 35/39 [00:06<00:00, 16.28it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 38/39 [00:07<00:00, 17.49it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 39/39 [00:07<00:00, 5.37it/s]
Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 27.53it/s]
(EngineCore pid=238247) INFO 11-01 06:18:21 [model_runner.py:1127] Graph capturing finished in 8 secs, took 0.33 GiB
(EngineCore pid=238247) INFO 11-01 06:18:22 [gpu_worker.py:759] Available KV cache memory: 60.89 GiB
(EngineCore pid=238247) INFO 11-01 06:18:22 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9155 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9245. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=238247) INFO 11-01 06:18:22 [kv_cache_utils.py:2327] GPU KV cache size: 1,794,641 tokens, Maximum concurrency for 32,768 tokens per request: 54.77x
(EngineCore pid=238247) INFO 11-01 06:18:22 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528)
(EngineCore pid=238247) INFO 11-01 06:18:22 [kv_cache_utils.py:854] kv lcm block sizes 528
(EngineCore pid=238247) INFO 11-01 06:18:22 [kernel_warmup.py:176] JIT kernel warmup starting.
(EngineCore pid=238247) INFO 11-01 06:18:22 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s.
(EngineCore pid=238247) INFO 11-01 06:18:22 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text.
(EngineCore pid=238247) INFO 11-01 06:18:23 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel.
Capturing CUDA graphs (PIECEWISE): 0%| | 0/39 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 3%|▎ | 1/39 [00:00<00:35, 1.08it/s] Capturing CUDA graphs (PIECEWISE): 5%|▌ | 2/39 [00:01<00:24, 1.49it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 3/39 [00:01<00:20, 1.79it/s] Capturing CUDA graphs (PIECEWISE): 10%|█ | 4/39 [00:02<00:17, 2.04it/s] Capturing CUDA graphs (PIECEWISE): 13%|█▎ | 5/39 [00:02<00:14, 2.33it/s] Capturing CUDA graphs (PIECEWISE): 15%|█▌ | 6/39 [00:02<00:12, 2.61it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 7/39 [00:03<00:11, 2.89it/s] Capturing CUDA graphs (PIECEWISE): 21%|██ | 8/39 [00:03<00:09, 3.18it/s] Capturing CUDA graphs (PIECEWISE): 23%|██▎ | 9/39 [00:03<00:08, 3.35it/s] Capturing CUDA graphs (PIECEWISE): 26%|██▌ | 10/39 [00:03<00:08, 3.49it/s] Capturing CUDA graphs (PIECEWISE): 28%|██▊ | 11/39 [00:04<00:07, 3.71it/s] Capturing CUDA graphs (PIECEWISE): 31%|███ | 12/39 [00:04<00:06, 4.03it/s] Capturing CUDA graphs (PIECEWISE): 33%|███▎ | 13/39 [00:04<00:05, 4.39it/s] Capturing CUDA graphs (PIECEWISE): 36%|███▌ | 14/39 [00:04<00:05, 4.86it/s] Capturing CUDA graphs (PIECEWISE): 38%|███▊ | 15/39 [00:04<00:04, 5.57it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 16/39 [00:04<00:03, 6.31it/s] Capturing CUDA graphs (PIECEWISE): 49%|████▊ | 19/39 [00:05<00:01, 10.54it/s] Capturing CUDA graphs (PIECEWISE): 56%|█████▋ | 22/39 [00:05<00:01, 13.11it/s] Capturing CUDA graphs (PIECEWISE): 62%|██████▏ | 24/39 [00:05<00:01, 14.58it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 26/39 [00:05<00:00, 15.72it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 29/39 [00:05<00:00, 17.74it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 32/39 [00:05<00:00, 19.14it/s] Capturing CUDA graphs (PIECEWISE): 90%|████████▉ | 35/39 [00:05<00:00, 20.12it/s] Capturing CUDA graphs (PIECEWISE): 97%|█████████▋| 38/39 [00:05<00:00, 20.71it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 39/39 [00:05<00:00, 6.54it/s]
Capturing CUDA graphs (FULL): 0%| | 0/23 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 13%|█▎ | 3/23 [00:00<00:00, 29.05it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 6/23 [00:00<00:00, 28.80it/s] Capturing CUDA graphs (FULL): 39%|███▉ | 9/23 [00:00<00:00, 28.69it/s] Capturing CUDA graphs (FULL): 52%|█████▏ | 12/23 [00:00<00:00, 28.17it/s] Capturing CUDA graphs (FULL): 65%|██████▌ | 15/23 [00:00<00:00, 27.72it/s] Capturing CUDA graphs (FULL): 78%|███████▊ | 18/23 [00:00<00:00, 27.93it/s] Capturing CUDA graphs (FULL): 91%|█████████▏| 21/23 [00:00<00:00, 28.06it/s] Capturing CUDA graphs (FULL): 100%|██████████| 23/23 [00:00<00:00, 28.19it/s]
(EngineCore pid=238247) INFO 11-01 06:18:50 [model_runner.py:1127] Graph capturing finished in 9 secs, took 0.18 GiB
(EngineCore pid=238247) INFO 11-01 06:18:50 [gpu_worker.py:971] CUDA graph pool memory: 0.18 GiB (actual), 0.35 GiB (estimated), difference: 0.17 GiB (92.6%).
(EngineCore pid=238247) INFO 11-01 06:18:50 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 10.43 GiB for consumed memory (weights + non-torch), 1.59 GiB for peak activation, and 0.18 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=65030718690` (60.56 GiB) to fit into requested memory, or `--kv-cache-memory=71394888192` (66.49 GiB) to fully utilize gpu memory. Current kv cache memory in use is 60.89 GiB.
(EngineCore pid=238247) INFO 11-01 06:18:51 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=238247) INFO 11-01 06:18:52 [core.py:395] init engine (profile, create kv cache, warmup model) took 156.63 s (compilation: 26.61 s)
(EngineCore pid=238247) INFO 11-01 06:18:52 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native'])
(MainProcess pid=237959) WARNING 11-01 06:18:53 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
(EngineCore pid=238247) WARNING 11-01 06:18:53 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
readout rl40: 5957 items in 113s
(EngineCore pid=238247) WARNING 11-01 06:20:46 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=238247) WARNING 11-01 06:20:47 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=238247) WARNING 11-01 06:20:48 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
progress {"time": "06:21:45", "elapsed_min": 1.0, "submitted": 256, "finished": 0, "in_flight": 256, "gen_tokens": 0, "tok_per_s": 0, "chain_tokens_median": 0.0, "chain_tokens_p90": 0.0, "chain_tokens_max": 0, "parse_rate": null, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:21:45 [loggers.py:335] Engine 000: Avg prompt throughput: 16421.3 tokens/s, Avg generation throughput: 1390.0 tokens/s, Running: 159 reqs, Waiting: 95 reqs, GPU KV cache usage: 29.6%, Prefix cache hit rate: 0.0%
progress {"time": "06:22:45", "elapsed_min": 2.0, "submitted": 452, "finished": 196, "in_flight": 256, "gen_tokens": 201165, "tok_per_s": 1676, "chain_tokens_median": 869.5, "chain_tokens_p90": 1822.5, "chain_tokens_max": 2575, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:22:45 [loggers.py:335] Engine 000: Avg prompt throughput: 5206.2 tokens/s, Avg generation throughput: 3585.2 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 32.4%, Prefix cache hit rate: 8.5%
progress {"time": "06:23:45", "elapsed_min": 3.0, "submitted": 553, "finished": 297, "in_flight": 256, "gen_tokens": 333447, "tok_per_s": 1852, "chain_tokens_median": 870.0, "chain_tokens_p90": 2090.4, "chain_tokens_max": 3888, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:23:45 [loggers.py:335] Engine 000: Avg prompt throughput: 3257.3 tokens/s, Avg generation throughput: 3707.7 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 35.2%, Prefix cache hit rate: 10.5%
progress {"time": "06:24:45", "elapsed_min": 4.0, "submitted": 649, "finished": 393, "in_flight": 256, "gen_tokens": 510241, "tok_per_s": 2126, "chain_tokens_median": 899.0, "chain_tokens_p90": 2829.8, "chain_tokens_max": 4987, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:24:45 [loggers.py:335] Engine 000: Avg prompt throughput: 4018.0 tokens/s, Avg generation throughput: 3548.3 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 36.4%, Prefix cache hit rate: 9.2%
progress {"time": "06:25:45", "elapsed_min": 5.0, "submitted": 754, "finished": 498, "in_flight": 256, "gen_tokens": 676799, "tok_per_s": 2256, "chain_tokens_median": 871.5, "chain_tokens_p90": 3164.6, "chain_tokens_max": 6387, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:25:45 [loggers.py:335] Engine 000: Avg prompt throughput: 4131.2 tokens/s, Avg generation throughput: 3290.2 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 37.5%, Prefix cache hit rate: 9.6%
progress {"time": "06:26:45", "elapsed_min": 6.0, "submitted": 857, "finished": 601, "in_flight": 256, "gen_tokens": 837777, "tok_per_s": 2327, "chain_tokens_median": 854.0, "chain_tokens_p90": 3328.0, "chain_tokens_max": 6663, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:26:45 [loggers.py:335] Engine 000: Avg prompt throughput: 4046.7 tokens/s, Avg generation throughput: 3250.7 tokens/s, Running: 159 reqs, Waiting: 96 reqs, GPU KV cache usage: 38.1%, Prefix cache hit rate: 7.5%
progress {"time": "06:27:45", "elapsed_min": 7.0, "submitted": 962, "finished": 706, "in_flight": 256, "gen_tokens": 1018545, "tok_per_s": 2425, "chain_tokens_median": 839.0, "chain_tokens_p90": 3385.5, "chain_tokens_max": 8569, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:27:45 [loggers.py:335] Engine 000: Avg prompt throughput: 4381.5 tokens/s, Avg generation throughput: 3201.9 tokens/s, Running: 159 reqs, Waiting: 96 reqs, GPU KV cache usage: 38.8%, Prefix cache hit rate: 4.5%
progress {"time": "06:28:45", "elapsed_min": 8.0, "submitted": 1062, "finished": 806, "in_flight": 256, "gen_tokens": 1189037, "tok_per_s": 2477, "chain_tokens_median": 852.0, "chain_tokens_p90": 3641.0, "chain_tokens_max": 8569, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:28:45 [loggers.py:335] Engine 000: Avg prompt throughput: 3912.5 tokens/s, Avg generation throughput: 3166.2 tokens/s, Running: 159 reqs, Waiting: 96 reqs, GPU KV cache usage: 39.4%, Prefix cache hit rate: 2.7%
progress {"time": "06:29:45", "elapsed_min": 9.0, "submitted": 1160, "finished": 904, "in_flight": 256, "gen_tokens": 1317132, "tok_per_s": 2439, "chain_tokens_median": 824.5, "chain_tokens_p90": 3486.7000000000007, "chain_tokens_max": 10200, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:29:45 [loggers.py:335] Engine 000: Avg prompt throughput: 3041.2 tokens/s, Avg generation throughput: 3385.7 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 42.6%, Prefix cache hit rate: 2.6%
progress {"time": "06:30:45", "elapsed_min": 10.0, "submitted": 1216, "finished": 960, "in_flight": 256, "gen_tokens": 1438008, "tok_per_s": 2396, "chain_tokens_median": 849.5, "chain_tokens_p90": 3694.3, "chain_tokens_max": 10215, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:30:45 [loggers.py:335] Engine 000: Avg prompt throughput: 2620.3 tokens/s, Avg generation throughput: 3374.8 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 44.9%, Prefix cache hit rate: 1.5%
progress {"time": "06:31:45", "elapsed_min": 11.0, "submitted": 1263, "finished": 1007, "in_flight": 256, "gen_tokens": 1563299, "tok_per_s": 2368, "chain_tokens_median": 878.0, "chain_tokens_p90": 3764.1999999999994, "chain_tokens_max": 11217, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:31:45 [loggers.py:335] Engine 000: Avg prompt throughput: 2856.0 tokens/s, Avg generation throughput: 3243.8 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 45.5%, Prefix cache hit rate: 1.6%
progress {"time": "06:32:45", "elapsed_min": 12.0, "submitted": 1356, "finished": 1100, "in_flight": 256, "gen_tokens": 1821858, "tok_per_s": 2530, "chain_tokens_median": 907.5, "chain_tokens_p90": 3874.5000000000005, "chain_tokens_max": 12785, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:32:45 [loggers.py:335] Engine 000: Avg prompt throughput: 5444.5 tokens/s, Avg generation throughput: 2540.4 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 44.2%, Prefix cache hit rate: 1.3%
progress {"time": "06:33:45", "elapsed_min": 13.0, "submitted": 1431, "finished": 1175, "in_flight": 256, "gen_tokens": 1990210, "tok_per_s": 2551, "chain_tokens_median": 938.0, "chain_tokens_p90": 3968.200000000003, "chain_tokens_max": 12785, "parse_rate": 0.9991489361702127, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:33:45 [loggers.py:335] Engine 000: Avg prompt throughput: 3702.6 tokens/s, Avg generation throughput: 2936.1 tokens/s, Running: 159 reqs, Waiting: 96 reqs, GPU KV cache usage: 45.6%, Prefix cache hit rate: 1.3%
progress {"time": "06:34:45", "elapsed_min": 14.0, "submitted": 1460, "finished": 1204, "in_flight": 256, "gen_tokens": 2065390, "tok_per_s": 2459, "chain_tokens_median": 949.0, "chain_tokens_p90": 4052.8, "chain_tokens_max": 12785, "parse_rate": 0.9991694352159468, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:34:45 [loggers.py:335] Engine 000: Avg prompt throughput: 1929.9 tokens/s, Avg generation throughput: 3410.3 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 48.0%, Prefix cache hit rate: 1.2%
progress {"time": "06:35:45", "elapsed_min": 15.0, "submitted": 1544, "finished": 1288, "in_flight": 256, "gen_tokens": 2267517, "tok_per_s": 2519, "chain_tokens_median": 1005.0, "chain_tokens_p90": 4078.3999999999983, "chain_tokens_max": 12785, "parse_rate": 0.9992236024844721, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:35:45 [loggers.py:335] Engine 000: Avg prompt throughput: 4280.5 tokens/s, Avg generation throughput: 2702.4 tokens/s, Running: 160 reqs, Waiting: 95 reqs, GPU KV cache usage: 49.2%, Prefix cache hit rate: 1.1%
progress {"time": "06:36:45", "elapsed_min": 16.0, "submitted": 1577, "finished": 1321, "in_flight": 256, "gen_tokens": 2351015, "tok_per_s": 2449, "chain_tokens_median": 1032.0, "chain_tokens_p90": 4113.0, "chain_tokens_max": 12785, "parse_rate": 0.9992429977289932, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:36:45 [loggers.py:335] Engine 000: Avg prompt throughput: 2119.6 tokens/s, Avg generation throughput: 3132.6 tokens/s, Running: 160 reqs, Waiting: 95 reqs, GPU KV cache usage: 49.5%, Prefix cache hit rate: 1.1%
progress {"time": "06:37:45", "elapsed_min": 17.0, "submitted": 1653, "finished": 1397, "in_flight": 256, "gen_tokens": 2569994, "tok_per_s": 2519, "chain_tokens_median": 1079.0, "chain_tokens_p90": 4253.200000000001, "chain_tokens_max": 12785, "parse_rate": 0.9992841803865425, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:37:45 [loggers.py:335] Engine 000: Avg prompt throughput: 4771.3 tokens/s, Avg generation throughput: 2446.2 tokens/s, Running: 160 reqs, Waiting: 95 reqs, GPU KV cache usage: 51.0%, Prefix cache hit rate: 0.9%
progress {"time": "06:38:45", "elapsed_min": 18.0, "submitted": 1715, "finished": 1459, "in_flight": 256, "gen_tokens": 2714150, "tok_per_s": 2513, "chain_tokens_median": 1098.0, "chain_tokens_p90": 4270.200000000001, "chain_tokens_max": 12785, "parse_rate": 0.9993145990404386, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:38:45 [loggers.py:335] Engine 000: Avg prompt throughput: 3590.4 tokens/s, Avg generation throughput: 2602.6 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 54.3%, Prefix cache hit rate: 0.9%
progress {"time": "06:39:45", "elapsed_min": 19.0, "submitted": 1755, "finished": 1499, "in_flight": 256, "gen_tokens": 2786569, "tok_per_s": 2444, "chain_tokens_median": 1101.0, "chain_tokens_p90": 4259.6, "chain_tokens_max": 12785, "parse_rate": 0.9993328885923949, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:39:45 [loggers.py:335] Engine 000: Avg prompt throughput: 2200.1 tokens/s, Avg generation throughput: 2868.5 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 56.2%, Prefix cache hit rate: 1.3%
progress {"time": "06:40:45", "elapsed_min": 20.0, "submitted": 1842, "finished": 1586, "in_flight": 256, "gen_tokens": 2991471, "tok_per_s": 2493, "chain_tokens_median": 1128.5, "chain_tokens_p90": 4361.5, "chain_tokens_max": 12785, "parse_rate": 0.9993694829760403, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:40:45 [loggers.py:335] Engine 000: Avg prompt throughput: 4530.8 tokens/s, Avg generation throughput: 2174.8 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 56.3%, Prefix cache hit rate: 1.9%
progress {"time": "06:41:45", "elapsed_min": 21.0, "submitted": 1887, "finished": 1631, "in_flight": 256, "gen_tokens": 3069446, "tok_per_s": 2436, "chain_tokens_median": 1128.0, "chain_tokens_p90": 4363.0, "chain_tokens_max": 12785, "parse_rate": 0.9993868792152054, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:41:45 [loggers.py:335] Engine 000: Avg prompt throughput: 2086.7 tokens/s, Avg generation throughput: 2751.1 tokens/s, Running: 160 reqs, Waiting: 95 reqs, GPU KV cache usage: 59.6%, Prefix cache hit rate: 1.9%
progress {"time": "06:42:45", "elapsed_min": 22.0, "submitted": 1954, "finished": 1698, "in_flight": 256, "gen_tokens": 3166957, "tok_per_s": 2399, "chain_tokens_median": 1114.0, "chain_tokens_p90": 4329.799999999999, "chain_tokens_max": 12785, "parse_rate": 0.9994110718492344, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:42:45 [loggers.py:335] Engine 000: Avg prompt throughput: 2616.8 tokens/s, Avg generation throughput: 2507.3 tokens/s, Running: 160 reqs, Waiting: 96 reqs, GPU KV cache usage: 61.7%, Prefix cache hit rate: 1.9%
progress {"time": "06:43:45", "elapsed_min": 23.0, "submitted": 2013, "finished": 1777, "in_flight": 236, "gen_tokens": 3309768, "tok_per_s": 2398, "chain_tokens_median": 1105.0, "chain_tokens_p90": 4342.000000000002, "chain_tokens_max": 12785, "parse_rate": 0.9994372537985369, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:43:45 [loggers.py:335] Engine 000: Avg prompt throughput: 3217.3 tokens/s, Avg generation throughput: 2219.1 tokens/s, Running: 160 reqs, Waiting: 76 reqs, GPU KV cache usage: 61.7%, Prefix cache hit rate: 2.7%
progress {"time": "06:44:45", "elapsed_min": 24.0, "submitted": 2013, "finished": 1812, "in_flight": 201, "gen_tokens": 3375158, "tok_per_s": 2344, "chain_tokens_median": 1102.0, "chain_tokens_p90": 4362.700000000001, "chain_tokens_max": 12785, "parse_rate": 0.9994481236203091, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:44:45 [loggers.py:335] Engine 000: Avg prompt throughput: 1573.5 tokens/s, Avg generation throughput: 2786.5 tokens/s, Running: 159 reqs, Waiting: 42 reqs, GPU KV cache usage: 66.8%, Prefix cache hit rate: 2.7%
progress {"time": "06:45:45", "elapsed_min": 25.0, "submitted": 2013, "finished": 1905, "in_flight": 108, "gen_tokens": 3549523, "tok_per_s": 2366, "chain_tokens_median": 1105.0, "chain_tokens_p90": 4387.6, "chain_tokens_max": 12785, "parse_rate": 0.9994750656167979, "looped": 0, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:45:45 [loggers.py:335] Engine 000: Avg prompt throughput: 3361.5 tokens/s, Avg generation throughput: 2003.8 tokens/s, Running: 108 reqs, Waiting: 0 reqs, GPU KV cache usage: 61.6%, Prefix cache hit rate: 3.0%
progress {"time": "06:46:45", "elapsed_min": 26.0, "submitted": 2013, "finished": 1928, "in_flight": 85, "gen_tokens": 3677386, "tok_per_s": 2357, "chain_tokens_median": 1123.5, "chain_tokens_p90": 4492.299999999999, "chain_tokens_max": 28457, "parse_rate": 0.9989626556016598, "looped": 1, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:46:45 [loggers.py:335] Engine 000: Avg prompt throughput: 1939.2 tokens/s, Avg generation throughput: 2013.4 tokens/s, Running: 85 reqs, Waiting: 0 reqs, GPU KV cache usage: 58.1%, Prefix cache hit rate: 2.9%
progress {"time": "06:47:45", "elapsed_min": 27.0, "submitted": 2013, "finished": 1950, "in_flight": 63, "gen_tokens": 3859932, "tok_per_s": 2383, "chain_tokens_median": 1147.5, "chain_tokens_p90": 4574.100000000001, "chain_tokens_max": 29867, "parse_rate": 0.9974358974358974, "looped": 4, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:47:45 [loggers.py:335] Engine 000: Avg prompt throughput: 1849.0 tokens/s, Avg generation throughput: 1739.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.7%, Prefix cache hit rate: 2.8%
progress {"time": "06:48:45", "elapsed_min": 28.0, "submitted": 2013, "finished": 1964, "in_flight": 49, "gen_tokens": 4056403, "tok_per_s": 2414, "chain_tokens_median": 1163.0, "chain_tokens_p90": 4644.6, "chain_tokens_max": 31570, "parse_rate": 0.995417515274949, "looped": 8, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:48:45 [loggers.py:335] Engine 000: Avg prompt throughput: 1257.6 tokens/s, Avg generation throughput: 1601.5 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.0%, Prefix cache hit rate: 2.7%
progress {"time": "06:49:45", "elapsed_min": 29.0, "submitted": 2013, "finished": 1968, "in_flight": 45, "gen_tokens": 4086569, "tok_per_s": 2348, "chain_tokens_median": 1164.5, "chain_tokens_p90": 4685.5, "chain_tokens_max": 31570, "parse_rate": 0.9954268292682927, "looped": 8, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:49:45 [loggers.py:335] Engine 000: Avg prompt throughput: 513.5 tokens/s, Avg generation throughput: 1453.5 tokens/s, Running: 45 reqs, Waiting: 0 reqs, GPU KV cache usage: 46.4%, Prefix cache hit rate: 2.6%
progress {"time": "06:50:45", "elapsed_min": 30.0, "submitted": 2013, "finished": 1977, "in_flight": 36, "gen_tokens": 4308316, "tok_per_s": 2393, "chain_tokens_median": 1168.0, "chain_tokens_p90": 4737.4000000000015, "chain_tokens_max": 32636, "parse_rate": 0.992412746585736, "looped": 14, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:50:45 [loggers.py:335] Engine 000: Avg prompt throughput: 478.6 tokens/s, Avg generation throughput: 1415.5 tokens/s, Running: 36 reqs, Waiting: 0 reqs, GPU KV cache usage: 38.6%, Prefix cache hit rate: 2.6%
progress {"time": "06:51:45", "elapsed_min": 31.0, "submitted": 2013, "finished": 1981, "in_flight": 32, "gen_tokens": 4419937, "tok_per_s": 2376, "chain_tokens_median": 1169.0, "chain_tokens_p90": 4785.0, "chain_tokens_max": 32636, "parse_rate": 0.9909136799596163, "looped": 17, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:51:45 [loggers.py:335] Engine 000: Avg prompt throughput: 235.8 tokens/s, Avg generation throughput: 1426.7 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.8%, Prefix cache hit rate: 2.6%
progress {"time": "06:52:45", "elapsed_min": 32.0, "submitted": 2013, "finished": 1982, "in_flight": 31, "gen_tokens": 4452525, "tok_per_s": 2319, "chain_tokens_median": 1171.0, "chain_tokens_p90": 4791.300000000001, "chain_tokens_max": 32636, "parse_rate": 0.9904137235116044, "looped": 18, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:52:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1342.0 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.1%, Prefix cache hit rate: 2.6%
progress {"time": "06:53:45", "elapsed_min": 33.0, "submitted": 2013, "finished": 1985, "in_flight": 28, "gen_tokens": 4549197, "tok_per_s": 2297, "chain_tokens_median": 1173.0, "chain_tokens_p90": 4816.000000000004, "chain_tokens_max": 32636, "parse_rate": 0.9889168765743073, "looped": 21, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:53:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1236.0 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.6%, Prefix cache hit rate: 2.6%
progress {"time": "06:54:45", "elapsed_min": 34.0, "submitted": 2013, "finished": 1988, "in_flight": 25, "gen_tokens": 4646412, "tok_per_s": 2278, "chain_tokens_median": 1178.5, "chain_tokens_p90": 4871.4, "chain_tokens_max": 32636, "parse_rate": 0.9874245472837022, "looped": 24, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:54:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1154.8 tokens/s, Running: 25 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.9%, Prefix cache hit rate: 2.6%
progress {"time": "06:55:45", "elapsed_min": 35.0, "submitted": 2013, "finished": 1995, "in_flight": 18, "gen_tokens": 4867637, "tok_per_s": 2318, "chain_tokens_median": 1185.0, "chain_tokens_p90": 4932.6, "chain_tokens_max": 32636, "parse_rate": 0.9839598997493735, "looped": 31, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:55:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1033.9 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.8%, Prefix cache hit rate: 2.6%
progress {"time": "06:56:45", "elapsed_min": 36.0, "submitted": 2013, "finished": 2003, "in_flight": 10, "gen_tokens": 5113461, "tok_per_s": 2367, "chain_tokens_median": 1189.0, "chain_tokens_p90": 4992.0, "chain_tokens_max": 32636, "parse_rate": 0.9800299550673989, "looped": 39, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:56:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 841.2 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 2.6%
progress {"time": "06:57:45", "elapsed_min": 37.0, "submitted": 2013, "finished": 2009, "in_flight": 4, "gen_tokens": 5306897, "tok_per_s": 2390, "chain_tokens_median": 1196.0, "chain_tokens_p90": 5066.0, "chain_tokens_max": 32636, "parse_rate": 0.9771030363364858, "looped": 45, "minutes_to_deadline": Infinity}
(MainProcess pid=237959) INFO 11-01 06:57:45 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 551.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 2.6%
reason rl40: {"submitted": 2013, "finished": 2013, "gen_tokens": 5436603, "t0": 1793514045.67846, "seconds": 2234.2781381607056, "tok_per_s": 2433.270463128418}
reason rl40: post-chain readout 1964 taken at generation, 0 by prefill
reason rl40: 2013 items, post-readout in 2235s total
latency rl40: {"readout_ms": {"median": 40.931111201643944, "p90": 72.46587350964546, "n": 100}, "reasoning_ms": {"median": 9934.432664886117, "p90": 40002.564517408624, "n": 20}}
(MainProcess pid=237959) INFO 11-01 07:04:02 [core_client.py:770] [shutdown] MPClient: start timeout=default
(MainProcess pid=237959) INFO 11-01 07:04:02 [core_client.py:772] [shutdown] MPClient: stopping engine manager
(MainProcess pid=237959) INFO 11-01 07:04:02 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(EngineCore pid=238247) INFO 11-01 07:04:02 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=238247) INFO 11-01 07:04:02 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=238247) INFO 11-01 07:04:02 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=238247) INFO 11-01 07:04:02 [core.py:1499] [shutdown] EngineCore: exiting busy loop
(MainProcess pid=237959) INFO 11-01 07:04:05 [core_client.py:774] [shutdown] MPClient: engine manager stopped
(MainProcess pid=237959) INFO 11-01 07:04:05 [core_client.py:775] [shutdown] MPClient: cleaning up background resources
(MainProcess pid=237959) INFO 11-01 07:04:05 [core_client.py:777] [shutdown] MPClient: complete

Xet Storage Details

Size:
40.9 kB
·
Xet hash:
576d53c8f1a771ca8938b575a9917da40ad2fcbf95b217b1a816f36687f54232

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.