Buckets:
| {"to_generate": 38825, "already_done": 3169, "minutes_left": 179.77852606376013} | |
| (MainProcess pid=64676) INFO 10-31 10:10:10 [model.py:733] Resolved architecture: Qwen3_5ForConditionalGeneration | |
| (MainProcess pid=64676) INFO 10-31 10:10:10 [model.py:2152] Using max model len 32768 | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1001.39it/s] | |
| (MainProcess pid=64676) INFO 10-31 10:10:14 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048. | |
| (MainProcess pid=64676) INFO 10-31 10:10:14 [config.py:688] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 6.39it/s] Parse safetensors files: 100%|██████████| 2/2 [00:00<00:00, 12.38it/s] | |
| (MainProcess pid=64676) INFO 10-31 10:10:14 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=64676) WARNING 10-31 10:10:27 [system_utils.py:153] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized | |
| (EngineCore pid=65100) INFO 10-31 10:10:37 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='Qwen/Qwen3.5-4B', speculative_config=None, tokenizer='Qwen/Qwen3.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=Qwen/Qwen3.5-4B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=65100) INFO 10-31 10:10:39 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_3bff52d0e3e64bcd9fc6f7189656b9d1 backend=nccl | |
| (EngineCore pid=65100) INFO 10-31 10:10:39 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=65100) INFO 10-31 10:10:39 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=65100) INFO 10-31 10:10:40 [model_runner.py:392] Loading model from scratch... | |
| Fetching 12 files: 0%| | 0/12 [00:00<?, ?it/s] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, 1007.14it/s] | |
| (EngineCore pid=65100) INFO 10-31 10:10:42 [cuda.py:565] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention | |
| (EngineCore pid=65100) INFO 10-31 10:10:42 [mm_encoder_attention.py:372] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention. | |
| (EngineCore pid=65100) INFO 10-31 10:10:42 [qwen_gdn_linear_attn.py:221] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). | |
| (EngineCore pid=65100) INFO 10-31 10:10:43 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=65100) INFO 10-31 10:10:43 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=65100) INFO 10-31 10:10:43 [base_loader.py:82] Initializing model took 2.418337 seconds | |
| (EngineCore pid=65100) INFO 10-31 10:10:44 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 8.68 GiB. Available RAM: 160.66 GiB. | |
| (EngineCore pid=65100) INFO 10-31 10:10:44 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.52it/s] | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.0.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.1.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.10.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.12.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.13.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.14.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.16.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.17.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.18.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.2.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.20.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.21.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.22.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.24.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.25.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.26.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.28.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.29.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.30.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.4.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.5.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.6.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.8.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| (EngineCore pid=65100) WARNING 10-31 10:10:45 [utils.py:333] Attempted to load weight layers.9.linear_attn.norm.weight with dtype torch.float32 into parameter with dtype torch.bfloat16 | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.52it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.52it/s] | |
| (EngineCore pid=65100) INFO 10-31 10:10:45 [default_loader.py:501] Loading weights took 0.80 seconds | |
| [transformers] Qwen3VL video processing does not apply the per-frame pixel cap the reference implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they would there. In v5.22 the capped behavior will become the default and `cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the reference behavior now, or `False` to keep the current behavior and silence this warning. | |
| (EngineCore pid=65100) INFO 10-31 10:10:47 [model_runner.py:424] Model loading took 8.56 GiB memory and 7.111196 seconds | |
| (EngineCore pid=65100) INFO 10-31 10:10:47 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=65100) INFO 10-31 10:10:47 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=65100) INFO 10-31 10:10:47 [interface.py:1096] Setting attention block size to 528 tokens to ensure that attention page size is >= mamba page size. | |
| (EngineCore pid=65100) INFO 10-31 10:10:47 [interface.py:1120] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal. | |
| (EngineCore pid=65100) INFO 10-31 10:10:47 [utils.py:347] Using LBNHC KV cache layout. | |
| (MainProcess pid=64676) INFO 10-31 10:10:48 [base.py:280] Multi-modal warmup completed in 20.938s | |
| (EngineCore pid=65100) INFO 10-31 10:10:55 [encoder_runner.py:131] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. | |
| (EngineCore pid=65100) INFO 10-31 10:10:59 [caching.py:342] reconstructed serializable fn from standalone compile artifacts. num_artifacts=13 num_submods=33 | |
| (EngineCore pid=65100) INFO 10-31 10:10:59 [decorators.py:319] [backbone] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/280d235bd2883e7a75f93a251c5ae942da381615d0d2c83834f39fb3bfac00c9/rank_0_0/model | |
| (EngineCore pid=65100) INFO 10-31 10:10:59 [monitor.py:62] [backbone] torch.compile took 0.92 s in total | |
| (EngineCore pid=65100) INFO 10-31 10:11:00 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.32 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:29, 1.71it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:00<00:22, 2.14it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:01<00:19, 2.41it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:01<00:19, 2.39it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:02<00:23, 1.99it/s] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:05<00:57, 1.28s/it] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:05<00:42, 1.05it/s] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:05<00:32, 1.34it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:06<00:25, 1.62it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:06<00:21, 1.94it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:06<00:17, 2.25it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:06<00:15, 2.56it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:07<00:13, 2.73it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:07<00:12, 2.92it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:07<00:11, 3.20it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:08<00:09, 3.51it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:08<00:06, 5.31it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:08<00:04, 7.15it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:08<00:03, 8.97it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:08<00:02, 10.47it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:08<00:02, 11.87it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:08<00:01, 13.25it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:08<00:01, 14.43it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:08<00:01, 15.49it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:09<00:01, 16.05it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:09<00:00, 16.75it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:09<00:00, 17.10it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:09<00:00, 17.40it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:09<00:00, 17.85it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:09<00:00, 18.18it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:09<00:00, 18.33it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:09<00:00, 18.47it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:09<00:00, 17.98it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:10<00:00, 5.08it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 24.92it/s] | |
| (EngineCore pid=65100) INFO 10-31 10:11:12 [model_runner.py:1127] Graph capturing finished in 11 secs, took 0.40 GiB | |
| (EngineCore pid=65100) INFO 10-31 10:11:12 [gpu_worker.py:759] Available KV cache memory: 64.21 GiB | |
| (EngineCore pid=65100) INFO 10-31 10:11:12 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9500 is equivalent to --gpu-memory-utilization=0.9440 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9560. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=65100) INFO 10-31 10:11:12 [kv_cache_utils.py:2327] GPU KV cache size: 1,892,470 tokens, Maximum concurrency for 32,768 tokens per request: 57.75x | |
| (EngineCore pid=65100) INFO 10-31 10:11:12 [kv_cache_utils.py:853] kv cache group sizes (528, 528, 528, 528) | |
| (EngineCore pid=65100) INFO 10-31 10:11:12 [kv_cache_utils.py:854] kv lcm block sizes 528 | |
| (EngineCore pid=65100) INFO 10-31 10:11:13 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=65100) INFO 10-31 10:11:13 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| (EngineCore pid=65100) INFO 10-31 10:11:13 [qwen_triton_warmup.py:296] Warming up Qwen GDN Triton kernels for model_type=qwen3_5_text. | |
| (EngineCore pid=65100) INFO 10-31 10:11:13 [qwen_vl_triton_warmup.py:57] Warmed position embedding and vision rotary kernels on grids=[(1, 16, 16), (1, 16, 2), (1, 2, 16), (1, 2, 2)]. | |
| (EngineCore pid=65100) INFO 10-31 10:11:13 [mamba_triton_warmup.py:42] Warmed Mamba batch_memcpy_kernel. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 2%|▏ | 1/51 [00:00<00:45, 1.09it/s] Capturing CUDA graphs (PIECEWISE): 4%|▍ | 2/51 [00:01<00:43, 1.12it/s] Capturing CUDA graphs (PIECEWISE): 6%|▌ | 3/51 [00:02<00:41, 1.17it/s] Capturing CUDA graphs (PIECEWISE): 8%|▊ | 4/51 [00:03<00:37, 1.24it/s] Capturing CUDA graphs (PIECEWISE): 10%|▉ | 5/51 [00:06<01:12, 1.58s/it] Capturing CUDA graphs (PIECEWISE): 12%|█▏ | 6/51 [00:06<00:56, 1.25s/it] Capturing CUDA graphs (PIECEWISE): 14%|█▎ | 7/51 [00:07<00:44, 1.02s/it] Capturing CUDA graphs (PIECEWISE): 16%|█▌ | 8/51 [00:07<00:36, 1.17it/s] Capturing CUDA graphs (PIECEWISE): 18%|█▊ | 9/51 [00:08<00:30, 1.39it/s] Capturing CUDA graphs (PIECEWISE): 20%|█▉ | 10/51 [00:08<00:25, 1.61it/s] Capturing CUDA graphs (PIECEWISE): 22%|██▏ | 11/51 [00:09<00:21, 1.87it/s] Capturing CUDA graphs (PIECEWISE): 24%|██▎ | 12/51 [00:09<00:17, 2.18it/s] Capturing CUDA graphs (PIECEWISE): 25%|██▌ | 13/51 [00:09<00:15, 2.40it/s] Capturing CUDA graphs (PIECEWISE): 27%|██▋ | 14/51 [00:09<00:13, 2.70it/s] Capturing CUDA graphs (PIECEWISE): 29%|██▉ | 15/51 [00:10<00:11, 3.04it/s] Capturing CUDA graphs (PIECEWISE): 31%|███▏ | 16/51 [00:10<00:09, 3.57it/s] Capturing CUDA graphs (PIECEWISE): 35%|███▌ | 18/51 [00:10<00:05, 5.51it/s] Capturing CUDA graphs (PIECEWISE): 39%|███▉ | 20/51 [00:10<00:04, 7.37it/s] Capturing CUDA graphs (PIECEWISE): 43%|████▎ | 22/51 [00:10<00:03, 9.06it/s] Capturing CUDA graphs (PIECEWISE): 47%|████▋ | 24/51 [00:10<00:02, 10.50it/s] Capturing CUDA graphs (PIECEWISE): 51%|█████ | 26/51 [00:11<00:02, 11.67it/s] Capturing CUDA graphs (PIECEWISE): 55%|█████▍ | 28/51 [00:11<00:01, 12.52it/s] Capturing CUDA graphs (PIECEWISE): 59%|█████▉ | 30/51 [00:11<00:01, 13.16it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 32/51 [00:11<00:01, 13.61it/s] Capturing CUDA graphs (PIECEWISE): 67%|██████▋ | 34/51 [00:11<00:01, 13.81it/s] Capturing CUDA graphs (PIECEWISE): 71%|███████ | 36/51 [00:11<00:01, 14.01it/s] Capturing CUDA graphs (PIECEWISE): 75%|███████▍ | 38/51 [00:11<00:00, 14.33it/s] Capturing CUDA graphs (PIECEWISE): 78%|███████▊ | 40/51 [00:11<00:00, 14.44it/s] Capturing CUDA graphs (PIECEWISE): 82%|████████▏ | 42/51 [00:12<00:00, 14.68it/s] Capturing CUDA graphs (PIECEWISE): 86%|████████▋ | 44/51 [00:12<00:00, 14.81it/s] Capturing CUDA graphs (PIECEWISE): 90%|█████████ | 46/51 [00:12<00:00, 14.93it/s] Capturing CUDA graphs (PIECEWISE): 94%|█████████▍| 48/51 [00:12<00:00, 14.94it/s] Capturing CUDA graphs (PIECEWISE): 98%|█████████▊| 50/51 [00:12<00:00, 14.87it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:12<00:00, 4.00it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/35 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 9%|▊ | 3/35 [00:00<00:01, 20.77it/s] Capturing CUDA graphs (FULL): 17%|█▋ | 6/35 [00:00<00:01, 21.03it/s] Capturing CUDA graphs (FULL): 26%|██▌ | 9/35 [00:00<00:01, 21.15it/s] Capturing CUDA graphs (FULL): 34%|███▍ | 12/35 [00:00<00:01, 21.21it/s] Capturing CUDA graphs (FULL): 43%|████▎ | 15/35 [00:00<00:00, 21.32it/s] Capturing CUDA graphs (FULL): 51%|█████▏ | 18/35 [00:00<00:00, 21.40it/s] Capturing CUDA graphs (FULL): 60%|██████ | 21/35 [00:00<00:00, 21.64it/s] Capturing CUDA graphs (FULL): 69%|██████▊ | 24/35 [00:01<00:00, 21.60it/s] Capturing CUDA graphs (FULL): 77%|███████▋ | 27/35 [00:01<00:00, 21.83it/s] Capturing CUDA graphs (FULL): 86%|████████▌ | 30/35 [00:01<00:00, 22.07it/s] Capturing CUDA graphs (FULL): 94%|█████████▍| 33/35 [00:01<00:00, 22.08it/s] Capturing CUDA graphs (FULL): 100%|██████████| 35/35 [00:01<00:00, 21.71it/s] | |
| (EngineCore pid=65100) INFO 10-31 10:11:28 [model_runner.py:1127] Graph capturing finished in 15 secs, took 0.24 GiB | |
| (EngineCore pid=65100) INFO 10-31 10:11:28 [gpu_worker.py:971] CUDA graph pool memory: 0.24 GiB (actual), 0.47 GiB (estimated), difference: 0.24 GiB (100.8%). | |
| (EngineCore pid=65100) INFO 10-31 10:11:28 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.95, 75.29 GiB). Actual usage is 9.0 GiB for consumed memory (weights + non-torch), 2.08 GiB for peak activation, and 0.24 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=68538206925` (63.83 GiB) to fit into requested memory, or `--kv-cache-memory=72349531648` (67.38 GiB) to fully utilize gpu memory. Current kv cache memory in use is 64.21 GiB. | |
| (EngineCore pid=65100) INFO 10-31 10:11:29 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=65100) INFO 10-31 10:11:30 [core.py:395] init engine (profile, create kv cache, warmup model) took 42.36 s (compilation: 0.92 s) | |
| Parse safetensors files: 0%| | 0/2 [00:00<?, ?it/s] Parse safetensors files: 50%|█████ | 1/2 [00:00<00:00, 4.77it/s] Parse safetensors files: 100%|██████████| 2/2 [00:01<00:00, 1.28it/s] Parse safetensors files: 100%|██████████| 2/2 [00:01<00:00, 1.44it/s] | |
| (EngineCore pid=65100) INFO 10-31 10:11:31 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=64676) WARNING 10-31 10:11:31 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=65100) WARNING 10-31 10:11:32 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=65100) WARNING 10-31 10:11:32 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=65100) WARNING 10-31 10:11:32 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| [rank0]:[W1031 10:11:38.406029396 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 243269632 bytes (free: 136183808, total: 85094825984). | |
| progress {"time": "10:12:30", "elapsed_min": 1.0, "submitted": 281, "finished": 25, "in_flight": 256, "gen_tokens": 60782, "tok_per_s": 1013, "chain_tokens_median": 588.0, "chain_tokens_p90": 837.0, "chain_tokens_max": 1238, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 177.4} | |
| (MainProcess pid=64676) INFO 10-31 10:12:30 [loggers.py:335] Engine 000: Avg prompt throughput: 2052.8 tokens/s, Avg generation throughput: 5538.4 tokens/s, Running: 256 reqs, Waiting: 729 reqs, GPU KV cache usage: 38.1%, Prefix cache hit rate: 45.8% | |
| progress {"time": "10:13:30", "elapsed_min": 2.0, "submitted": 308, "finished": 52, "in_flight": 256, "gen_tokens": 157913, "tok_per_s": 1316, "chain_tokens_median": 686.0, "chain_tokens_p90": 1226.7, "chain_tokens_max": 2541, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 176.4} | |
| (MainProcess pid=64676) INFO 10-31 10:13:30 [loggers.py:335] Engine 000: Avg prompt throughput: 480.3 tokens/s, Avg generation throughput: 5672.5 tokens/s, Running: 256 reqs, Waiting: 731 reqs, GPU KV cache usage: 48.9%, Prefix cache hit rate: 43.8% | |
| progress {"time": "10:14:30", "elapsed_min": 3.0, "submitted": 324, "finished": 68, "in_flight": 256, "gen_tokens": 261540, "tok_per_s": 1453, "chain_tokens_median": 729.5, "chain_tokens_p90": 1793.3000000000004, "chain_tokens_max": 3825, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 175.4} | |
| (MainProcess pid=64676) INFO 10-31 10:14:30 [loggers.py:335] Engine 000: Avg prompt throughput: 427.5 tokens/s, Avg generation throughput: 5229.1 tokens/s, Running: 256 reqs, Waiting: 712 reqs, GPU KV cache usage: 56.2%, Prefix cache hit rate: 43.1% | |
| progress {"time": "10:15:30", "elapsed_min": 4.0, "submitted": 342, "finished": 86, "in_flight": 256, "gen_tokens": 401877, "tok_per_s": 1674, "chain_tokens_median": 771.0, "chain_tokens_p90": 2735.7, "chain_tokens_max": 4980, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 174.4} | |
| (MainProcess pid=64676) INFO 10-31 10:15:30 [loggers.py:335] Engine 000: Avg prompt throughput: 585.8 tokens/s, Avg generation throughput: 4955.8 tokens/s, Running: 256 reqs, Waiting: 691 reqs, GPU KV cache usage: 59.8%, Prefix cache hit rate: 42.6% | |
| progress {"time": "10:16:30", "elapsed_min": 5.0, "submitted": 361, "finished": 105, "in_flight": 256, "gen_tokens": 622731, "tok_per_s": 2076, "chain_tokens_median": 836.5, "chain_tokens_p90": 3732.4, "chain_tokens_max": 6066, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 173.4} | |
| (MainProcess pid=64676) INFO 10-31 10:16:30 [loggers.py:335] Engine 000: Avg prompt throughput: 604.0 tokens/s, Avg generation throughput: 4947.4 tokens/s, Running: 256 reqs, Waiting: 670 reqs, GPU KV cache usage: 58.4%, Prefix cache hit rate: 40.9% | |
| progress {"time": "10:17:30", "elapsed_min": 6.0, "submitted": 394, "finished": 138, "in_flight": 256, "gen_tokens": 969780, "tok_per_s": 2694, "chain_tokens_median": 871.5, "chain_tokens_p90": 4598.500000000001, "chain_tokens_max": 7134, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 172.4} | |
| (MainProcess pid=64676) INFO 10-31 10:17:30 [loggers.py:335] Engine 000: Avg prompt throughput: 750.7 tokens/s, Avg generation throughput: 4839.6 tokens/s, Running: 256 reqs, Waiting: 669 reqs, GPU KV cache usage: 56.8%, Prefix cache hit rate: 41.1% | |
| progress {"time": "10:18:30", "elapsed_min": 7.0, "submitted": 418, "finished": 162, "in_flight": 256, "gen_tokens": 1283614, "tok_per_s": 3056, "chain_tokens_median": 975.5, "chain_tokens_p90": 5030.900000000001, "chain_tokens_max": 8168, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 171.4} | |
| (MainProcess pid=64676) INFO 10-31 10:18:30 [loggers.py:335] Engine 000: Avg prompt throughput: 392.7 tokens/s, Avg generation throughput: 4652.7 tokens/s, Running: 256 reqs, Waiting: 670 reqs, GPU KV cache usage: 54.2%, Prefix cache hit rate: 41.3% | |
| progress {"time": "10:19:30", "elapsed_min": 8.0, "submitted": 441, "finished": 185, "in_flight": 256, "gen_tokens": 1571354, "tok_per_s": 3273, "chain_tokens_median": 1069.5, "chain_tokens_p90": 5145.3, "chain_tokens_max": 9128, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 170.4} | |
| (MainProcess pid=64676) INFO 10-31 10:19:30 [loggers.py:335] Engine 000: Avg prompt throughput: 298.1 tokens/s, Avg generation throughput: 4440.1 tokens/s, Running: 256 reqs, Waiting: 686 reqs, GPU KV cache usage: 55.7%, Prefix cache hit rate: 40.5% | |
| progress {"time": "10:20:30", "elapsed_min": 9.0, "submitted": 465, "finished": 209, "in_flight": 256, "gen_tokens": 1775460, "tok_per_s": 3288, "chain_tokens_median": 1073.0, "chain_tokens_p90": 5135.5, "chain_tokens_max": 9298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 169.4} | |
| (MainProcess pid=64676) INFO 10-31 10:20:30 [loggers.py:335] Engine 000: Avg prompt throughput: 466.0 tokens/s, Avg generation throughput: 4993.9 tokens/s, Running: 255 reqs, Waiting: 673 reqs, GPU KV cache usage: 59.1%, Prefix cache hit rate: 40.5% | |
| progress {"time": "10:21:30", "elapsed_min": 10.0, "submitted": 487, "finished": 231, "in_flight": 256, "gen_tokens": 1980416, "tok_per_s": 3301, "chain_tokens_median": 1091.0, "chain_tokens_p90": 5102.5, "chain_tokens_max": 9298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 168.4} | |
| (MainProcess pid=64676) INFO 10-31 10:21:30 [loggers.py:335] Engine 000: Avg prompt throughput: 493.5 tokens/s, Avg generation throughput: 4900.5 tokens/s, Running: 256 reqs, Waiting: 661 reqs, GPU KV cache usage: 60.3%, Prefix cache hit rate: 36.5% | |
| progress {"time": "10:22:30", "elapsed_min": 11.0, "submitted": 509, "finished": 253, "in_flight": 256, "gen_tokens": 2227355, "tok_per_s": 3375, "chain_tokens_median": 1134.0, "chain_tokens_p90": 5207.4, "chain_tokens_max": 11421, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 167.4} | |
| (MainProcess pid=64676) INFO 10-31 10:22:30 [loggers.py:335] Engine 000: Avg prompt throughput: 510.0 tokens/s, Avg generation throughput: 4827.8 tokens/s, Running: 256 reqs, Waiting: 664 reqs, GPU KV cache usage: 62.3%, Prefix cache hit rate: 38.5% | |
| progress {"time": "10:23:30", "elapsed_min": 12.0, "submitted": 529, "finished": 273, "in_flight": 256, "gen_tokens": 2480429, "tok_per_s": 3445, "chain_tokens_median": 1233.5, "chain_tokens_p90": 5298.7, "chain_tokens_max": 11421, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 166.4} | |
| (MainProcess pid=64676) INFO 10-31 10:23:30 [loggers.py:335] Engine 000: Avg prompt throughput: 356.1 tokens/s, Avg generation throughput: 4828.4 tokens/s, Running: 256 reqs, Waiting: 671 reqs, GPU KV cache usage: 64.4%, Prefix cache hit rate: 36.8% | |
| progress {"time": "10:24:30", "elapsed_min": 13.0, "submitted": 550, "finished": 294, "in_flight": 256, "gen_tokens": 2691235, "tok_per_s": 3450, "chain_tokens_median": 1270.0, "chain_tokens_p90": 5327.0, "chain_tokens_max": 11421, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 165.4} | |
| (MainProcess pid=64676) INFO 10-31 10:24:30 [loggers.py:335] Engine 000: Avg prompt throughput: 451.0 tokens/s, Avg generation throughput: 4635.4 tokens/s, Running: 255 reqs, Waiting: 660 reqs, GPU KV cache usage: 64.3%, Prefix cache hit rate: 37.7% | |
| progress {"time": "10:25:30", "elapsed_min": 14.0, "submitted": 568, "finished": 312, "in_flight": 256, "gen_tokens": 2881711, "tok_per_s": 3430, "chain_tokens_median": 1301.5, "chain_tokens_p90": 5370.499999999999, "chain_tokens_max": 11421, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 164.4} | |
| (MainProcess pid=64676) INFO 10-31 10:25:30 [loggers.py:335] Engine 000: Avg prompt throughput: 454.7 tokens/s, Avg generation throughput: 4554.7 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 64.0%, Prefix cache hit rate: 38.2% | |
| progress {"time": "10:26:30", "elapsed_min": 15.0, "submitted": 595, "finished": 339, "in_flight": 256, "gen_tokens": 3193491, "tok_per_s": 3548, "chain_tokens_median": 1388.5, "chain_tokens_p90": 5441.5, "chain_tokens_max": 11804, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 163.4} | |
| (MainProcess pid=64676) INFO 10-31 10:26:30 [loggers.py:335] Engine 000: Avg prompt throughput: 386.0 tokens/s, Avg generation throughput: 4567.8 tokens/s, Running: 256 reqs, Waiting: 654 reqs, GPU KV cache usage: 65.2%, Prefix cache hit rate: 38.5% | |
| progress {"time": "10:27:30", "elapsed_min": 16.0, "submitted": 611, "finished": 355, "in_flight": 256, "gen_tokens": 3391647, "tok_per_s": 3533, "chain_tokens_median": 1430.0, "chain_tokens_p90": 5470.500000000001, "chain_tokens_max": 11804, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 162.4} | |
| (MainProcess pid=64676) INFO 10-31 10:27:30 [loggers.py:335] Engine 000: Avg prompt throughput: 365.9 tokens/s, Avg generation throughput: 4546.8 tokens/s, Running: 255 reqs, Waiting: 646 reqs, GPU KV cache usage: 68.2%, Prefix cache hit rate: 39.0% | |
| progress {"time": "10:28:30", "elapsed_min": 17.0, "submitted": 633, "finished": 377, "in_flight": 256, "gen_tokens": 3703616, "tok_per_s": 3631, "chain_tokens_median": 1470.5, "chain_tokens_p90": 5706.499999999999, "chain_tokens_max": 12298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 161.4} | |
| (MainProcess pid=64676) INFO 10-31 10:28:30 [loggers.py:335] Engine 000: Avg prompt throughput: 369.5 tokens/s, Avg generation throughput: 4401.4 tokens/s, Running: 256 reqs, Waiting: 651 reqs, GPU KV cache usage: 68.1%, Prefix cache hit rate: 40.6% | |
| progress {"time": "10:29:30", "elapsed_min": 18.0, "submitted": 656, "finished": 400, "in_flight": 256, "gen_tokens": 3918812, "tok_per_s": 3628, "chain_tokens_median": 1460.0, "chain_tokens_p90": 5686.400000000001, "chain_tokens_max": 12298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 160.4} | |
| (MainProcess pid=64676) INFO 10-31 10:29:30 [loggers.py:335] Engine 000: Avg prompt throughput: 339.6 tokens/s, Avg generation throughput: 4375.6 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 67.7%, Prefix cache hit rate: 39.7% | |
| progress {"time": "10:30:30", "elapsed_min": 19.0, "submitted": 677, "finished": 421, "in_flight": 256, "gen_tokens": 4183933, "tok_per_s": 3670, "chain_tokens_median": 1523.0, "chain_tokens_p90": 5698.400000000001, "chain_tokens_max": 12298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 159.4} | |
| (MainProcess pid=64676) INFO 10-31 10:30:30 [loggers.py:335] Engine 000: Avg prompt throughput: 330.4 tokens/s, Avg generation throughput: 4426.9 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 67.9%, Prefix cache hit rate: 39.3% | |
| progress {"time": "10:31:30", "elapsed_min": 20.0, "submitted": 699, "finished": 443, "in_flight": 256, "gen_tokens": 4369034, "tok_per_s": 3641, "chain_tokens_median": 1510.0, "chain_tokens_p90": 5677.9000000000015, "chain_tokens_max": 12298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 158.4} | |
| (MainProcess pid=64676) INFO 10-31 10:31:30 [loggers.py:335] Engine 000: Avg prompt throughput: 431.8 tokens/s, Avg generation throughput: 4328.7 tokens/s, Running: 256 reqs, Waiting: 650 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 39.8% | |
| progress {"time": "10:32:30", "elapsed_min": 21.0, "submitted": 717, "finished": 461, "in_flight": 256, "gen_tokens": 4532316, "tok_per_s": 3597, "chain_tokens_median": 1506.5, "chain_tokens_p90": 5675.700000000001, "chain_tokens_max": 12298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 157.4} | |
| (MainProcess pid=64676) INFO 10-31 10:32:30 [loggers.py:335] Engine 000: Avg prompt throughput: 357.6 tokens/s, Avg generation throughput: 4290.8 tokens/s, Running: 256 reqs, Waiting: 650 reqs, GPU KV cache usage: 74.1%, Prefix cache hit rate: 39.8% | |
| progress {"time": "10:33:30", "elapsed_min": 22.0, "submitted": 732, "finished": 476, "in_flight": 256, "gen_tokens": 4726577, "tok_per_s": 3581, "chain_tokens_median": 1517.0, "chain_tokens_p90": 5719.8, "chain_tokens_max": 12298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 156.4} | |
| (MainProcess pid=64676) INFO 10-31 10:33:30 [loggers.py:335] Engine 000: Avg prompt throughput: 279.1 tokens/s, Avg generation throughput: 4222.8 tokens/s, Running: 256 reqs, Waiting: 654 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 39.7% | |
| progress {"time": "10:34:30", "elapsed_min": 23.0, "submitted": 743, "finished": 487, "in_flight": 256, "gen_tokens": 4885972, "tok_per_s": 3540, "chain_tokens_median": 1551.5, "chain_tokens_p90": 5731.2, "chain_tokens_max": 12298, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 155.4} | |
| (MainProcess pid=64676) INFO 10-31 10:34:30 [loggers.py:335] Engine 000: Avg prompt throughput: 217.1 tokens/s, Avg generation throughput: 4154.7 tokens/s, Running: 255 reqs, Waiting: 645 reqs, GPU KV cache usage: 79.5%, Prefix cache hit rate: 38.7% | |
| progress {"time": "10:35:30", "elapsed_min": 24.0, "submitted": 758, "finished": 502, "in_flight": 256, "gen_tokens": 5037478, "tok_per_s": 3498, "chain_tokens_median": 1542.0, "chain_tokens_p90": 5723.7, "chain_tokens_max": 16994, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 154.4} | |
| (MainProcess pid=64676) INFO 10-31 10:35:30 [loggers.py:335] Engine 000: Avg prompt throughput: 349.2 tokens/s, Avg generation throughput: 3966.4 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 80.5%, Prefix cache hit rate: 37.4% | |
| progress {"time": "10:36:30", "elapsed_min": 25.0, "submitted": 783, "finished": 527, "in_flight": 256, "gen_tokens": 5360170, "tok_per_s": 3573, "chain_tokens_median": 1549.5, "chain_tokens_p90": 5781.799999999999, "chain_tokens_max": 16994, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 153.4} | |
| (MainProcess pid=64676) INFO 10-31 10:36:30 [loggers.py:335] Engine 000: Avg prompt throughput: 449.2 tokens/s, Avg generation throughput: 3842.5 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 37.6% | |
| progress {"time": "10:37:30", "elapsed_min": 26.0, "submitted": 797, "finished": 541, "in_flight": 256, "gen_tokens": 5560804, "tok_per_s": 3564, "chain_tokens_median": 1566.0, "chain_tokens_p90": 5866.5, "chain_tokens_max": 16994, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 152.4} | |
| (MainProcess pid=64676) INFO 10-31 10:37:30 [loggers.py:335] Engine 000: Avg prompt throughput: 363.9 tokens/s, Avg generation throughput: 3958.1 tokens/s, Running: 256 reqs, Waiting: 634 reqs, GPU KV cache usage: 82.2%, Prefix cache hit rate: 38.5% | |
| progress {"time": "10:38:30", "elapsed_min": 27.0, "submitted": 816, "finished": 560, "in_flight": 256, "gen_tokens": 5785035, "tok_per_s": 3571, "chain_tokens_median": 1623.5, "chain_tokens_p90": 5840.500000000004, "chain_tokens_max": 16994, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 151.4} | |
| (MainProcess pid=64676) INFO 10-31 10:38:30 [loggers.py:335] Engine 000: Avg prompt throughput: 476.6 tokens/s, Avg generation throughput: 3791.3 tokens/s, Running: 256 reqs, Waiting: 628 reqs, GPU KV cache usage: 82.2%, Prefix cache hit rate: 39.6% | |
| progress {"time": "10:39:30", "elapsed_min": 28.0, "submitted": 839, "finished": 583, "in_flight": 256, "gen_tokens": 6005010, "tok_per_s": 3574, "chain_tokens_median": 1615.0, "chain_tokens_p90": 5860.500000000002, "chain_tokens_max": 16994, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 150.4} | |
| (MainProcess pid=64676) INFO 10-31 10:39:30 [loggers.py:335] Engine 000: Avg prompt throughput: 455.7 tokens/s, Avg generation throughput: 3778.3 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 81.1%, Prefix cache hit rate: 40.2% | |
| progress {"time": "10:40:30", "elapsed_min": 29.0, "submitted": 857, "finished": 601, "in_flight": 256, "gen_tokens": 6207446, "tok_per_s": 3567, "chain_tokens_median": 1623.5, "chain_tokens_p90": 5877.500000000002, "chain_tokens_max": 16994, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 149.4} | |
| (MainProcess pid=64676) INFO 10-31 10:40:30 [loggers.py:335] Engine 000: Avg prompt throughput: 424.1 tokens/s, Avg generation throughput: 3761.3 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 80.5%, Prefix cache hit rate: 38.8% | |
| progress {"time": "10:41:30", "elapsed_min": 30.0, "submitted": 873, "finished": 617, "in_flight": 256, "gen_tokens": 6340903, "tok_per_s": 3523, "chain_tokens_median": 1625.0, "chain_tokens_p90": 5822.400000000003, "chain_tokens_max": 16994, "parse_rate": 1.0, "looped": 0, "minutes_to_deadline": 148.4} | |
| (MainProcess pid=64676) INFO 10-31 10:41:30 [loggers.py:335] Engine 000: Avg prompt throughput: 251.4 tokens/s, Avg generation throughput: 3915.6 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 37.9% | |
| progress {"time": "10:42:30", "elapsed_min": 31.0, "submitted": 889, "finished": 633, "in_flight": 256, "gen_tokens": 6573423, "tok_per_s": 3534, "chain_tokens_median": 1646.0, "chain_tokens_p90": 5837.700000000001, "chain_tokens_max": 32359, "parse_rate": 0.9996050552922591, "looped": 1, "minutes_to_deadline": 147.4} | |
| (MainProcess pid=64676) INFO 10-31 10:42:30 [loggers.py:335] Engine 000: Avg prompt throughput: 359.6 tokens/s, Avg generation throughput: 3693.3 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 86.9%, Prefix cache hit rate: 38.8% | |
| progress {"time": "10:43:30", "elapsed_min": 32.0, "submitted": 908, "finished": 652, "in_flight": 256, "gen_tokens": 6903680, "tok_per_s": 3595, "chain_tokens_median": 1649.5, "chain_tokens_p90": 5918.100000000001, "chain_tokens_max": 32674, "parse_rate": 0.9984662576687117, "looped": 4, "minutes_to_deadline": 146.4} | |
| (MainProcess pid=64676) INFO 10-31 10:43:30 [loggers.py:335] Engine 000: Avg prompt throughput: 311.8 tokens/s, Avg generation throughput: 3796.0 tokens/s, Running: 256 reqs, Waiting: 642 reqs, GPU KV cache usage: 85.5%, Prefix cache hit rate: 37.9% | |
| progress {"time": "10:44:30", "elapsed_min": 33.0, "submitted": 924, "finished": 668, "in_flight": 256, "gen_tokens": 7100273, "tok_per_s": 3586, "chain_tokens_median": 1650.0, "chain_tokens_p90": 5916.0, "chain_tokens_max": 32674, "parse_rate": 0.9981287425149701, "looped": 5, "minutes_to_deadline": 145.4} | |
| (MainProcess pid=64676) INFO 10-31 10:44:30 [loggers.py:335] Engine 000: Avg prompt throughput: 397.0 tokens/s, Avg generation throughput: 3753.4 tokens/s, Running: 256 reqs, Waiting: 643 reqs, GPU KV cache usage: 87.0%, Prefix cache hit rate: 38.8% | |
| progress {"time": "10:45:30", "elapsed_min": 34.0, "submitted": 939, "finished": 683, "in_flight": 256, "gen_tokens": 7219177, "tok_per_s": 3539, "chain_tokens_median": 1648.5, "chain_tokens_p90": 5879.9, "chain_tokens_max": 32674, "parse_rate": 0.9981698389458272, "looped": 5, "minutes_to_deadline": 144.4} | |
| (MainProcess pid=64676) INFO 10-31 10:45:30 [loggers.py:335] Engine 000: Avg prompt throughput: 282.8 tokens/s, Avg generation throughput: 3710.7 tokens/s, Running: 256 reqs, Waiting: 644 reqs, GPU KV cache usage: 89.1%, Prefix cache hit rate: 39.3% | |
| progress {"time": "10:46:30", "elapsed_min": 35.0, "submitted": 955, "finished": 699, "in_flight": 256, "gen_tokens": 7434816, "tok_per_s": 3540, "chain_tokens_median": 1649.5, "chain_tokens_p90": 5879.5, "chain_tokens_max": 32674, "parse_rate": 0.9978540772532188, "looped": 6, "minutes_to_deadline": 143.4} | |
| (MainProcess pid=64676) INFO 10-31 10:46:30 [loggers.py:335] Engine 000: Avg prompt throughput: 341.7 tokens/s, Avg generation throughput: 3676.4 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 87.0%, Prefix cache hit rate: 38.3% | |
| progress {"time": "10:47:30", "elapsed_min": 36.0, "submitted": 974, "finished": 718, "in_flight": 256, "gen_tokens": 7634263, "tok_per_s": 3534, "chain_tokens_median": 1657.0, "chain_tokens_p90": 5873.6, "chain_tokens_max": 32674, "parse_rate": 0.9979108635097493, "looped": 6, "minutes_to_deadline": 142.4} | |
| (MainProcess pid=64676) INFO 10-31 10:47:30 [loggers.py:335] Engine 000: Avg prompt throughput: 307.7 tokens/s, Avg generation throughput: 3654.7 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 88.3%, Prefix cache hit rate: 38.8% | |
| progress {"time": "10:48:30", "elapsed_min": 37.0, "submitted": 988, "finished": 732, "in_flight": 256, "gen_tokens": 7811346, "tok_per_s": 3518, "chain_tokens_median": 1667.0, "chain_tokens_p90": 5887.300000000002, "chain_tokens_max": 32674, "parse_rate": 0.9979508196721312, "looped": 6, "minutes_to_deadline": 141.4} | |
| (MainProcess pid=64676) INFO 10-31 10:48:30 [loggers.py:335] Engine 000: Avg prompt throughput: 303.0 tokens/s, Avg generation throughput: 3680.7 tokens/s, Running: 256 reqs, Waiting: 634 reqs, GPU KV cache usage: 89.9%, Prefix cache hit rate: 39.6% | |
| progress {"time": "10:49:30", "elapsed_min": 38.0, "submitted": 1003, "finished": 747, "in_flight": 256, "gen_tokens": 8067515, "tok_per_s": 3538, "chain_tokens_median": 1678.5, "chain_tokens_p90": 5910.4000000000015, "chain_tokens_max": 32674, "parse_rate": 0.9969879518072289, "looped": 9, "minutes_to_deadline": 140.4} | |
| (MainProcess pid=64676) INFO 10-31 10:49:30 [loggers.py:335] Engine 000: Avg prompt throughput: 372.5 tokens/s, Avg generation throughput: 3667.9 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 87.5%, Prefix cache hit rate: 40.1% | |
| progress {"time": "10:50:30", "elapsed_min": 39.0, "submitted": 1023, "finished": 767, "in_flight": 256, "gen_tokens": 8487184, "tok_per_s": 3627, "chain_tokens_median": 1681.5, "chain_tokens_p90": 5973.3, "chain_tokens_max": 32674, "parse_rate": 0.9951108213820078, "looped": 15, "minutes_to_deadline": 139.4} | |
| (MainProcess pid=64676) INFO 10-31 10:50:30 [loggers.py:335] Engine 000: Avg prompt throughput: 513.4 tokens/s, Avg generation throughput: 3624.8 tokens/s, Running: 255 reqs, Waiting: 617 reqs, GPU KV cache usage: 76.8%, Prefix cache hit rate: 41.3% | |
| progress {"time": "10:51:30", "elapsed_min": 40.0, "submitted": 1044, "finished": 788, "in_flight": 256, "gen_tokens": 8721583, "tok_per_s": 3634, "chain_tokens_median": 1684.5, "chain_tokens_p90": 5967.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9949238578680203, "looped": 16, "minutes_to_deadline": 138.4} | |
| (MainProcess pid=64676) INFO 10-31 10:51:30 [loggers.py:335] Engine 000: Avg prompt throughput: 380.4 tokens/s, Avg generation throughput: 3906.7 tokens/s, Running: 256 reqs, Waiting: 627 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 40.9% | |
| progress {"time": "10:52:30", "elapsed_min": 41.0, "submitted": 1062, "finished": 806, "in_flight": 256, "gen_tokens": 8888523, "tok_per_s": 3613, "chain_tokens_median": 1681.5, "chain_tokens_p90": 5938.800000000001, "chain_tokens_max": 32674, "parse_rate": 0.9950372208436724, "looped": 16, "minutes_to_deadline": 137.4} | |
| (MainProcess pid=64676) INFO 10-31 10:52:30 [loggers.py:335] Engine 000: Avg prompt throughput: 422.8 tokens/s, Avg generation throughput: 3795.6 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 82.0%, Prefix cache hit rate: 41.2% | |
| progress {"time": "10:53:30", "elapsed_min": 42.0, "submitted": 1078, "finished": 822, "in_flight": 256, "gen_tokens": 9099629, "tok_per_s": 3611, "chain_tokens_median": 1689.5, "chain_tokens_p90": 5930.500000000001, "chain_tokens_max": 32674, "parse_rate": 0.9948296836982968, "looped": 17, "minutes_to_deadline": 136.4} | |
| (MainProcess pid=64676) INFO 10-31 10:53:30 [loggers.py:335] Engine 000: Avg prompt throughput: 290.4 tokens/s, Avg generation throughput: 3864.5 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 84.5%, Prefix cache hit rate: 41.2% | |
| progress {"time": "10:54:30", "elapsed_min": 43.0, "submitted": 1090, "finished": 834, "in_flight": 256, "gen_tokens": 9304048, "tok_per_s": 3606, "chain_tokens_median": 1702.0, "chain_tokens_p90": 5960.5, "chain_tokens_max": 32674, "parse_rate": 0.9946043165467626, "looped": 18, "minutes_to_deadline": 135.4} | |
| (MainProcess pid=64676) INFO 10-31 10:54:30 [loggers.py:335] Engine 000: Avg prompt throughput: 238.8 tokens/s, Avg generation throughput: 3804.6 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 39.9% | |
| progress {"time": "10:55:30", "elapsed_min": 44.0, "submitted": 1104, "finished": 848, "in_flight": 256, "gen_tokens": 9565418, "tok_per_s": 3623, "chain_tokens_median": 1703.0, "chain_tokens_p90": 5987.8, "chain_tokens_max": 32674, "parse_rate": 0.9938089622641509, "looped": 21, "minutes_to_deadline": 134.4} | |
| (MainProcess pid=64676) INFO 10-31 10:55:30 [loggers.py:335] Engine 000: Avg prompt throughput: 286.0 tokens/s, Avg generation throughput: 3838.9 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 84.8%, Prefix cache hit rate: 39.4% | |
| progress {"time": "10:56:30", "elapsed_min": 45.0, "submitted": 1119, "finished": 863, "in_flight": 256, "gen_tokens": 9768607, "tok_per_s": 3618, "chain_tokens_median": 1723.0, "chain_tokens_p90": 5998.200000000001, "chain_tokens_max": 32674, "parse_rate": 0.9939165701042874, "looped": 21, "minutes_to_deadline": 133.4} | |
| (MainProcess pid=64676) INFO 10-31 10:56:30 [loggers.py:335] Engine 000: Avg prompt throughput: 258.0 tokens/s, Avg generation throughput: 3770.3 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 84.7%, Prefix cache hit rate: 40.1% | |
| progress {"time": "10:57:30", "elapsed_min": 46.0, "submitted": 1135, "finished": 879, "in_flight": 256, "gen_tokens": 9915669, "tok_per_s": 3592, "chain_tokens_median": 1700.5, "chain_tokens_p90": 5987.0, "chain_tokens_max": 32674, "parse_rate": 0.9937428896473265, "looped": 22, "minutes_to_deadline": 132.4} | |
| (MainProcess pid=64676) INFO 10-31 10:57:30 [loggers.py:335] Engine 000: Avg prompt throughput: 367.6 tokens/s, Avg generation throughput: 3628.7 tokens/s, Running: 256 reqs, Waiting: 606 reqs, GPU KV cache usage: 82.0%, Prefix cache hit rate: 38.0% | |
| progress {"time": "10:58:30", "elapsed_min": 47.0, "submitted": 1157, "finished": 901, "in_flight": 256, "gen_tokens": 10210875, "tok_per_s": 3621, "chain_tokens_median": 1693.5, "chain_tokens_p90": 6052.500000000002, "chain_tokens_max": 32674, "parse_rate": 0.9936182019977803, "looped": 23, "minutes_to_deadline": 131.4} | |
| (MainProcess pid=64676) INFO 10-31 10:58:30 [loggers.py:335] Engine 000: Avg prompt throughput: 487.3 tokens/s, Avg generation throughput: 3744.1 tokens/s, Running: 256 reqs, Waiting: 609 reqs, GPU KV cache usage: 78.8%, Prefix cache hit rate: 40.8% | |
| progress {"time": "10:59:30", "elapsed_min": 48.0, "submitted": 1171, "finished": 915, "in_flight": 256, "gen_tokens": 10413036, "tok_per_s": 3615, "chain_tokens_median": 1723.0, "chain_tokens_p90": 6091.5, "chain_tokens_max": 32674, "parse_rate": 0.9937158469945355, "looped": 23, "minutes_to_deadline": 130.4} | |
| (MainProcess pid=64676) INFO 10-31 10:59:30 [loggers.py:335] Engine 000: Avg prompt throughput: 216.6 tokens/s, Avg generation throughput: 4009.2 tokens/s, Running: 256 reqs, Waiting: 607 reqs, GPU KV cache usage: 80.9%, Prefix cache hit rate: 39.9% | |
| progress {"time": "11:00:30", "elapsed_min": 49.0, "submitted": 1191, "finished": 935, "in_flight": 256, "gen_tokens": 10605433, "tok_per_s": 3607, "chain_tokens_median": 1719.0, "chain_tokens_p90": 6046.299999999999, "chain_tokens_max": 32674, "parse_rate": 0.9938502673796792, "looped": 23, "minutes_to_deadline": 129.4} | |
| (MainProcess pid=64676) INFO 10-31 11:00:30 [loggers.py:335] Engine 000: Avg prompt throughput: 432.0 tokens/s, Avg generation throughput: 3825.6 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 83.5%, Prefix cache hit rate: 39.9% | |
| progress {"time": "11:01:30", "elapsed_min": 50.0, "submitted": 1201, "finished": 945, "in_flight": 256, "gen_tokens": 10717393, "tok_per_s": 3572, "chain_tokens_median": 1713.0, "chain_tokens_p90": 6055.199999999999, "chain_tokens_max": 32674, "parse_rate": 0.9939153439153439, "looped": 23, "minutes_to_deadline": 128.4} | |
| (MainProcess pid=64676) INFO 10-31 11:01:30 [loggers.py:335] Engine 000: Avg prompt throughput: 234.0 tokens/s, Avg generation throughput: 3795.8 tokens/s, Running: 256 reqs, Waiting: 602 reqs, GPU KV cache usage: 85.8%, Prefix cache hit rate: 40.7% | |
| progress {"time": "11:02:30", "elapsed_min": 51.0, "submitted": 1217, "finished": 961, "in_flight": 256, "gen_tokens": 10999544, "tok_per_s": 3594, "chain_tokens_median": 1721.5, "chain_tokens_p90": 6119.300000000003, "chain_tokens_max": 32674, "parse_rate": 0.9937565036420395, "looped": 24, "minutes_to_deadline": 127.4} | |
| (MainProcess pid=64676) INFO 10-31 11:02:30 [loggers.py:335] Engine 000: Avg prompt throughput: 345.6 tokens/s, Avg generation throughput: 3774.4 tokens/s, Running: 256 reqs, Waiting: 594 reqs, GPU KV cache usage: 83.9%, Prefix cache hit rate: 40.1% | |
| progress {"time": "11:03:30", "elapsed_min": 52.0, "submitted": 1237, "finished": 981, "in_flight": 256, "gen_tokens": 11232257, "tok_per_s": 3600, "chain_tokens_median": 1730.0, "chain_tokens_p90": 6112.1, "chain_tokens_max": 32674, "parse_rate": 0.9938837920489296, "looped": 24, "minutes_to_deadline": 126.4} | |
| (MainProcess pid=64676) INFO 10-31 11:03:30 [loggers.py:335] Engine 000: Avg prompt throughput: 320.7 tokens/s, Avg generation throughput: 3796.0 tokens/s, Running: 256 reqs, Waiting: 608 reqs, GPU KV cache usage: 83.5%, Prefix cache hit rate: 40.7% | |
| progress {"time": "11:04:30", "elapsed_min": 53.0, "submitted": 1254, "finished": 998, "in_flight": 256, "gen_tokens": 11518558, "tok_per_s": 3622, "chain_tokens_median": 1748.5, "chain_tokens_p90": 6143.5, "chain_tokens_max": 32674, "parse_rate": 0.9934869739478958, "looped": 26, "minutes_to_deadline": 125.4} | |
| (MainProcess pid=64676) INFO 10-31 11:04:30 [loggers.py:335] Engine 000: Avg prompt throughput: 297.2 tokens/s, Avg generation throughput: 3830.0 tokens/s, Running: 256 reqs, Waiting: 615 reqs, GPU KV cache usage: 83.6%, Prefix cache hit rate: 40.9% | |
| progress {"time": "11:05:30", "elapsed_min": 54.0, "submitted": 1267, "finished": 1011, "in_flight": 256, "gen_tokens": 11651615, "tok_per_s": 3596, "chain_tokens_median": 1733.5, "chain_tokens_p90": 6142.500000000002, "chain_tokens_max": 32674, "parse_rate": 0.993570722057369, "looped": 26, "minutes_to_deadline": 124.4} | |
| (MainProcess pid=64676) INFO 10-31 11:05:30 [loggers.py:335] Engine 000: Avg prompt throughput: 291.5 tokens/s, Avg generation throughput: 3761.7 tokens/s, Running: 256 reqs, Waiting: 601 reqs, GPU KV cache usage: 87.3%, Prefix cache hit rate: 39.1% | |
| progress {"time": "11:06:30", "elapsed_min": 55.0, "submitted": 1289, "finished": 1033, "in_flight": 256, "gen_tokens": 11935302, "tok_per_s": 3617, "chain_tokens_median": 1742.5, "chain_tokens_p90": 6125.700000000001, "chain_tokens_max": 32674, "parse_rate": 0.9932236205227493, "looped": 28, "minutes_to_deadline": 123.4} | |
| (MainProcess pid=64676) INFO 10-31 11:06:30 [loggers.py:335] Engine 000: Avg prompt throughput: 534.4 tokens/s, Avg generation throughput: 3582.1 tokens/s, Running: 256 reqs, Waiting: 611 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 40.3% | |
| progress {"time": "11:07:30", "elapsed_min": 56.0, "submitted": 1305, "finished": 1049, "in_flight": 256, "gen_tokens": 12090273, "tok_per_s": 3598, "chain_tokens_median": 1743.5, "chain_tokens_p90": 6118.0, "chain_tokens_max": 32674, "parse_rate": 0.9933269780743565, "looped": 28, "minutes_to_deadline": 122.4} | |
| (MainProcess pid=64676) INFO 10-31 11:07:30 [loggers.py:335] Engine 000: Avg prompt throughput: 446.7 tokens/s, Avg generation throughput: 3633.4 tokens/s, Running: 256 reqs, Waiting: 602 reqs, GPU KV cache usage: 88.0%, Prefix cache hit rate: 40.7% | |
| progress {"time": "11:08:30", "elapsed_min": 57.0, "submitted": 1327, "finished": 1071, "in_flight": 256, "gen_tokens": 12435295, "tok_per_s": 3636, "chain_tokens_median": 1746.0, "chain_tokens_p90": 6148.400000000001, "chain_tokens_max": 32674, "parse_rate": 0.9929971988795518, "looped": 30, "minutes_to_deadline": 121.4} | |
| (MainProcess pid=64676) INFO 10-31 11:08:30 [loggers.py:335] Engine 000: Avg prompt throughput: 289.6 tokens/s, Avg generation throughput: 3770.4 tokens/s, Running: 255 reqs, Waiting: 627 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 39.9% | |
| progress {"time": "11:09:30", "elapsed_min": 58.0, "submitted": 1340, "finished": 1084, "in_flight": 256, "gen_tokens": 12602317, "tok_per_s": 3621, "chain_tokens_median": 1750.5, "chain_tokens_p90": 6158.5, "chain_tokens_max": 32674, "parse_rate": 0.9930811808118081, "looped": 30, "minutes_to_deadline": 120.4} | |
| (MainProcess pid=64676) INFO 10-31 11:09:30 [loggers.py:335] Engine 000: Avg prompt throughput: 416.6 tokens/s, Avg generation throughput: 3719.0 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 88.7%, Prefix cache hit rate: 41.2% | |
| progress {"time": "11:10:30", "elapsed_min": 59.0, "submitted": 1348, "finished": 1092, "in_flight": 256, "gen_tokens": 12741181, "tok_per_s": 3599, "chain_tokens_median": 1762.5, "chain_tokens_p90": 6173.600000000002, "chain_tokens_max": 32674, "parse_rate": 0.9931318681318682, "looped": 30, "minutes_to_deadline": 119.4} | |
| (MainProcess pid=64676) INFO 10-31 11:10:30 [loggers.py:335] Engine 000: Avg prompt throughput: 174.1 tokens/s, Avg generation throughput: 3822.1 tokens/s, Running: 256 reqs, Waiting: 618 reqs, GPU KV cache usage: 93.9%, Prefix cache hit rate: 41.9% | |
| progress {"time": "11:11:30", "elapsed_min": 60.0, "submitted": 1359, "finished": 1103, "in_flight": 256, "gen_tokens": 12874734, "tok_per_s": 3576, "chain_tokens_median": 1753.0, "chain_tokens_p90": 6167.5, "chain_tokens_max": 32674, "parse_rate": 0.992973708068903, "looped": 31, "minutes_to_deadline": 118.4} | |
| (MainProcess pid=64676) INFO 10-31 11:11:30 [loggers.py:335] Engine 000: Avg prompt throughput: 312.4 tokens/s, Avg generation throughput: 3591.2 tokens/s, Running: 256 reqs, Waiting: 605 reqs, GPU KV cache usage: 92.6%, Prefix cache hit rate: 41.9% | |
| progress {"time": "11:12:30", "elapsed_min": 61.0, "submitted": 1375, "finished": 1119, "in_flight": 256, "gen_tokens": 13162988, "tok_per_s": 3596, "chain_tokens_median": 1765.5, "chain_tokens_p90": 6182.5, "chain_tokens_max": 32674, "parse_rate": 0.9924039320822162, "looped": 34, "minutes_to_deadline": 117.4} | |
| (MainProcess pid=64676) INFO 10-31 11:12:30 [loggers.py:335] Engine 000: Avg prompt throughput: 371.1 tokens/s, Avg generation throughput: 3544.0 tokens/s, Running: 256 reqs, Waiting: 598 reqs, GPU KV cache usage: 88.3%, Prefix cache hit rate: 42.2% | |
| progress {"time": "11:13:30", "elapsed_min": 62.0, "submitted": 1394, "finished": 1139, "in_flight": 255, "gen_tokens": 13606825, "tok_per_s": 3658, "chain_tokens_median": 1788.0, "chain_tokens_p90": 6225.0, "chain_tokens_max": 32674, "parse_rate": 0.9916593503072871, "looped": 38, "minutes_to_deadline": 116.4} | |
| (MainProcess pid=64676) INFO 10-31 11:13:30 [loggers.py:335] Engine 000: Avg prompt throughput: 302.0 tokens/s, Avg generation throughput: 3727.4 tokens/s, Running: 255 reqs, Waiting: 610 reqs, GPU KV cache usage: 83.2%, Prefix cache hit rate: 42.5% | |
| progress {"time": "11:14:30", "elapsed_min": 63.0, "submitted": 1411, "finished": 1155, "in_flight": 256, "gen_tokens": 13804915, "tok_per_s": 3652, "chain_tokens_median": 1786.0, "chain_tokens_p90": 6233.800000000003, "chain_tokens_max": 32674, "parse_rate": 0.9915584415584415, "looped": 39, "minutes_to_deadline": 115.4} | |
| (MainProcess pid=64676) INFO 10-31 11:14:30 [loggers.py:335] Engine 000: Avg prompt throughput: 295.5 tokens/s, Avg generation throughput: 3800.0 tokens/s, Running: 256 reqs, Waiting: 612 reqs, GPU KV cache usage: 83.5%, Prefix cache hit rate: 41.0% | |
| progress {"time": "11:15:30", "elapsed_min": 64.0, "submitted": 1435, "finished": 1179, "in_flight": 256, "gen_tokens": 14243786, "tok_per_s": 3709, "chain_tokens_median": 1819.0, "chain_tokens_p90": 6257.0, "chain_tokens_max": 32674, "parse_rate": 0.9906700593723494, "looped": 44, "minutes_to_deadline": 114.4} | |
| (MainProcess pid=64676) INFO 10-31 11:15:30 [loggers.py:335] Engine 000: Avg prompt throughput: 370.7 tokens/s, Avg generation throughput: 3791.2 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 77.9%, Prefix cache hit rate: 43.4% | |
| progress {"time": "11:16:30", "elapsed_min": 65.0, "submitted": 1451, "finished": 1195, "in_flight": 256, "gen_tokens": 14417926, "tok_per_s": 3697, "chain_tokens_median": 1809.5, "chain_tokens_p90": 6269.000000000004, "chain_tokens_max": 32674, "parse_rate": 0.9907949790794979, "looped": 44, "minutes_to_deadline": 113.4} | |
| (MainProcess pid=64676) INFO 10-31 11:16:30 [loggers.py:335] Engine 000: Avg prompt throughput: 469.9 tokens/s, Avg generation throughput: 3846.6 tokens/s, Running: 256 reqs, Waiting: 626 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 44.1% | |
| progress {"time": "11:17:30", "elapsed_min": 66.0, "submitted": 1466, "finished": 1210, "in_flight": 256, "gen_tokens": 14626856, "tok_per_s": 3693, "chain_tokens_median": 1820.5, "chain_tokens_p90": 6278.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9907024793388429, "looped": 45, "minutes_to_deadline": 112.4} | |
| (MainProcess pid=64676) INFO 10-31 11:17:30 [loggers.py:335] Engine 000: Avg prompt throughput: 278.9 tokens/s, Avg generation throughput: 4009.2 tokens/s, Running: 256 reqs, Waiting: 633 reqs, GPU KV cache usage: 78.1%, Prefix cache hit rate: 43.8% | |
| progress {"time": "11:18:30", "elapsed_min": 67.0, "submitted": 1483, "finished": 1227, "in_flight": 256, "gen_tokens": 14853565, "tok_per_s": 3695, "chain_tokens_median": 1823.5, "chain_tokens_p90": 6298.3, "chain_tokens_max": 32674, "parse_rate": 0.9908312958435208, "looped": 45, "minutes_to_deadline": 111.4} | |
| (MainProcess pid=64676) INFO 10-31 11:18:30 [loggers.py:335] Engine 000: Avg prompt throughput: 371.8 tokens/s, Avg generation throughput: 3902.0 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 78.7%, Prefix cache hit rate: 43.7% | |
| progress {"time": "11:19:30", "elapsed_min": 68.0, "submitted": 1502, "finished": 1246, "in_flight": 256, "gen_tokens": 15040204, "tok_per_s": 3686, "chain_tokens_median": 1812.0, "chain_tokens_p90": 6302.499999999999, "chain_tokens_max": 32674, "parse_rate": 0.9909711075441413, "looped": 45, "minutes_to_deadline": 110.4} | |
| (MainProcess pid=64676) INFO 10-31 11:19:30 [loggers.py:335] Engine 000: Avg prompt throughput: 411.2 tokens/s, Avg generation throughput: 3833.7 tokens/s, Running: 256 reqs, Waiting: 612 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 43.8% | |
| progress {"time": "11:20:30", "elapsed_min": 69.0, "submitted": 1528, "finished": 1272, "in_flight": 256, "gen_tokens": 15272202, "tok_per_s": 3689, "chain_tokens_median": 1786.0, "chain_tokens_p90": 6271.000000000002, "chain_tokens_max": 32674, "parse_rate": 0.9911556603773585, "looped": 45, "minutes_to_deadline": 109.4} | |
| (MainProcess pid=64676) INFO 10-31 11:20:30 [loggers.py:335] Engine 000: Avg prompt throughput: 526.9 tokens/s, Avg generation throughput: 3782.2 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 78.5%, Prefix cache hit rate: 43.4% | |
| progress {"time": "11:21:30", "elapsed_min": 70.0, "submitted": 1550, "finished": 1294, "in_flight": 256, "gen_tokens": 15467096, "tok_per_s": 3682, "chain_tokens_median": 1765.5, "chain_tokens_p90": 6246.0, "chain_tokens_max": 32674, "parse_rate": 0.991306027820711, "looped": 45, "minutes_to_deadline": 108.4} | |
| (MainProcess pid=64676) INFO 10-31 11:21:30 [loggers.py:335] Engine 000: Avg prompt throughput: 342.3 tokens/s, Avg generation throughput: 3851.0 tokens/s, Running: 254 reqs, Waiting: 624 reqs, GPU KV cache usage: 80.6%, Prefix cache hit rate: 41.5% | |
| progress {"time": "11:22:30", "elapsed_min": 71.0, "submitted": 1563, "finished": 1307, "in_flight": 256, "gen_tokens": 15690545, "tok_per_s": 3683, "chain_tokens_median": 1769.0, "chain_tokens_p90": 6271.000000000002, "chain_tokens_max": 32674, "parse_rate": 0.9910099464422342, "looped": 47, "minutes_to_deadline": 107.4} | |
| (MainProcess pid=64676) INFO 10-31 11:22:30 [loggers.py:335] Engine 000: Avg prompt throughput: 368.3 tokens/s, Avg generation throughput: 3876.9 tokens/s, Running: 256 reqs, Waiting: 612 reqs, GPU KV cache usage: 80.9%, Prefix cache hit rate: 40.3% | |
| progress {"time": "11:23:30", "elapsed_min": 72.0, "submitted": 1585, "finished": 1329, "in_flight": 256, "gen_tokens": 15889513, "tok_per_s": 3678, "chain_tokens_median": 1777.5, "chain_tokens_p90": 6246.0, "chain_tokens_max": 32674, "parse_rate": 0.9911587659894657, "looped": 47, "minutes_to_deadline": 106.4} | |
| (MainProcess pid=64676) INFO 10-31 11:23:30 [loggers.py:335] Engine 000: Avg prompt throughput: 465.0 tokens/s, Avg generation throughput: 3735.8 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 81.8%, Prefix cache hit rate: 40.4% | |
| progress {"time": "11:24:30", "elapsed_min": 73.0, "submitted": 1602, "finished": 1346, "in_flight": 256, "gen_tokens": 16104956, "tok_per_s": 3677, "chain_tokens_median": 1762.5, "chain_tokens_p90": 6243.799999999999, "chain_tokens_max": 32674, "parse_rate": 0.9908989598811293, "looped": 49, "minutes_to_deadline": 105.4} | |
| (MainProcess pid=64676) INFO 10-31 11:24:30 [loggers.py:335] Engine 000: Avg prompt throughput: 275.0 tokens/s, Avg generation throughput: 3880.8 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 81.8%, Prefix cache hit rate: 40.0% | |
| progress {"time": "11:25:30", "elapsed_min": 74.0, "submitted": 1616, "finished": 1360, "in_flight": 256, "gen_tokens": 16236789, "tok_per_s": 3657, "chain_tokens_median": 1753.0, "chain_tokens_p90": 6230.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9909926470588235, "looped": 49, "minutes_to_deadline": 104.4} | |
| (MainProcess pid=64676) INFO 10-31 11:25:30 [loggers.py:335] Engine 000: Avg prompt throughput: 322.5 tokens/s, Avg generation throughput: 3480.2 tokens/s, Running: 256 reqs, Waiting: 613 reqs, GPU KV cache usage: 82.3%, Prefix cache hit rate: 38.9% | |
| progress {"time": "11:26:30", "elapsed_min": 75.0, "submitted": 1638, "finished": 1382, "in_flight": 256, "gen_tokens": 16559977, "tok_per_s": 3680, "chain_tokens_median": 1769.5, "chain_tokens_p90": 6245.6, "chain_tokens_max": 32674, "parse_rate": 0.9909551374819102, "looped": 50, "minutes_to_deadline": 103.4} | |
| (MainProcess pid=64676) INFO 10-31 11:26:30 [loggers.py:335] Engine 000: Avg prompt throughput: 308.7 tokens/s, Avg generation throughput: 3522.9 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 82.3%, Prefix cache hit rate: 38.3% | |
| progress {"time": "11:27:30", "elapsed_min": 76.0, "submitted": 1650, "finished": 1394, "in_flight": 256, "gen_tokens": 16696751, "tok_per_s": 3661, "chain_tokens_median": 1762.0, "chain_tokens_p90": 6243.0, "chain_tokens_max": 32674, "parse_rate": 0.9908536585365854, "looped": 51, "minutes_to_deadline": 102.4} | |
| (MainProcess pid=64676) INFO 10-31 11:27:30 [loggers.py:335] Engine 000: Avg prompt throughput: 306.7 tokens/s, Avg generation throughput: 3394.7 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 83.2%, Prefix cache hit rate: 38.3% | |
| progress {"time": "11:28:30", "elapsed_min": 77.0, "submitted": 1668, "finished": 1412, "in_flight": 256, "gen_tokens": 16900167, "tok_per_s": 3658, "chain_tokens_median": 1751.0, "chain_tokens_p90": 6235.4000000000015, "chain_tokens_max": 32674, "parse_rate": 0.9907932011331445, "looped": 52, "minutes_to_deadline": 101.4} | |
| (MainProcess pid=64676) INFO 10-31 11:28:30 [loggers.py:335] Engine 000: Avg prompt throughput: 340.9 tokens/s, Avg generation throughput: 3177.2 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 82.4%, Prefix cache hit rate: 38.9% | |
| progress {"time": "11:29:30", "elapsed_min": 78.0, "submitted": 1684, "finished": 1428, "in_flight": 256, "gen_tokens": 17134549, "tok_per_s": 3661, "chain_tokens_median": 1762.5, "chain_tokens_p90": 6263.600000000002, "chain_tokens_max": 32674, "parse_rate": 0.9908963585434174, "looped": 52, "minutes_to_deadline": 100.4} | |
| (MainProcess pid=64676) INFO 10-31 11:29:30 [loggers.py:335] Engine 000: Avg prompt throughput: 241.8 tokens/s, Avg generation throughput: 3625.2 tokens/s, Running: 256 reqs, Waiting: 627 reqs, GPU KV cache usage: 82.8%, Prefix cache hit rate: 36.2% | |
| progress {"time": "11:30:30", "elapsed_min": 79.0, "submitted": 1704, "finished": 1448, "in_flight": 256, "gen_tokens": 17372205, "tok_per_s": 3665, "chain_tokens_median": 1761.0, "chain_tokens_p90": 6265.800000000001, "chain_tokens_max": 32674, "parse_rate": 0.9908494475138122, "looped": 53, "minutes_to_deadline": 99.4} | |
| (MainProcess pid=64676) INFO 10-31 11:30:30 [loggers.py:335] Engine 000: Avg prompt throughput: 480.8 tokens/s, Avg generation throughput: 3675.8 tokens/s, Running: 255 reqs, Waiting: 628 reqs, GPU KV cache usage: 82.7%, Prefix cache hit rate: 35.3% | |
| progress {"time": "11:31:30", "elapsed_min": 80.0, "submitted": 1717, "finished": 1461, "in_flight": 256, "gen_tokens": 17517071, "tok_per_s": 3649, "chain_tokens_median": 1760.0, "chain_tokens_p90": 6267.4, "chain_tokens_max": 32674, "parse_rate": 0.990930869267625, "looped": 53, "minutes_to_deadline": 98.4} | |
| (MainProcess pid=64676) INFO 10-31 11:31:30 [loggers.py:335] Engine 000: Avg prompt throughput: 314.0 tokens/s, Avg generation throughput: 2955.4 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 83.1%, Prefix cache hit rate: 35.8% | |
| progress {"time": "11:32:30", "elapsed_min": 81.0, "submitted": 1732, "finished": 1476, "in_flight": 256, "gen_tokens": 17701260, "tok_per_s": 3642, "chain_tokens_median": 1759.5, "chain_tokens_p90": 6267.4, "chain_tokens_max": 32674, "parse_rate": 0.9910230352303523, "looped": 53, "minutes_to_deadline": 97.4} | |
| (MainProcess pid=64676) INFO 10-31 11:32:30 [loggers.py:335] Engine 000: Avg prompt throughput: 274.9 tokens/s, Avg generation throughput: 3283.8 tokens/s, Running: 256 reqs, Waiting: 614 reqs, GPU KV cache usage: 83.0%, Prefix cache hit rate: 35.8% | |
| progress {"time": "11:33:30", "elapsed_min": 82.0, "submitted": 1748, "finished": 1492, "in_flight": 256, "gen_tokens": 17955596, "tok_per_s": 3649, "chain_tokens_median": 1761.0, "chain_tokens_p90": 6287.500000000003, "chain_tokens_max": 32674, "parse_rate": 0.99078418230563, "looped": 55, "minutes_to_deadline": 96.4} | |
| (MainProcess pid=64676) INFO 10-31 11:33:30 [loggers.py:335] Engine 000: Avg prompt throughput: 327.6 tokens/s, Avg generation throughput: 3842.6 tokens/s, Running: 256 reqs, Waiting: 609 reqs, GPU KV cache usage: 81.9%, Prefix cache hit rate: 35.1% | |
| progress {"time": "11:34:30", "elapsed_min": 83.0, "submitted": 1769, "finished": 1513, "in_flight": 256, "gen_tokens": 18277313, "tok_per_s": 3670, "chain_tokens_median": 1770.5, "chain_tokens_p90": 6298.900000000001, "chain_tokens_max": 32674, "parse_rate": 0.9905816259087905, "looped": 57, "minutes_to_deadline": 95.4} | |
| (MainProcess pid=64676) INFO 10-31 11:34:30 [loggers.py:335] Engine 000: Avg prompt throughput: 384.3 tokens/s, Avg generation throughput: 3765.8 tokens/s, Running: 256 reqs, Waiting: 618 reqs, GPU KV cache usage: 81.4%, Prefix cache hit rate: 35.7% | |
| progress {"time": "11:35:30", "elapsed_min": 84.0, "submitted": 1793, "finished": 1537, "in_flight": 256, "gen_tokens": 18640003, "tok_per_s": 3698, "chain_tokens_median": 1775.0, "chain_tokens_p90": 6305.0, "chain_tokens_max": 32674, "parse_rate": 0.9902407286922577, "looped": 60, "minutes_to_deadline": 94.4} | |
| (MainProcess pid=64676) INFO 10-31 11:35:30 [loggers.py:335] Engine 000: Avg prompt throughput: 531.7 tokens/s, Avg generation throughput: 3778.6 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 78.9%, Prefix cache hit rate: 35.5% | |
| progress {"time": "11:36:30", "elapsed_min": 85.0, "submitted": 1813, "finished": 1557, "in_flight": 256, "gen_tokens": 18843323, "tok_per_s": 3695, "chain_tokens_median": 1765.5, "chain_tokens_p90": 6284.800000000001, "chain_tokens_max": 32674, "parse_rate": 0.9902055234425177, "looped": 61, "minutes_to_deadline": 93.4} | |
| (MainProcess pid=64676) INFO 10-31 11:36:30 [loggers.py:335] Engine 000: Avg prompt throughput: 380.6 tokens/s, Avg generation throughput: 3902.3 tokens/s, Running: 256 reqs, Waiting: 639 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 36.8% | |
| progress {"time": "11:37:30", "elapsed_min": 86.0, "submitted": 1830, "finished": 1574, "in_flight": 256, "gen_tokens": 19044744, "tok_per_s": 3691, "chain_tokens_median": 1767.5, "chain_tokens_p90": 6282.5, "chain_tokens_max": 32674, "parse_rate": 0.9903113087674714, "looped": 61, "minutes_to_deadline": 92.4} | |
| (MainProcess pid=64676) INFO 10-31 11:37:30 [loggers.py:335] Engine 000: Avg prompt throughput: 302.8 tokens/s, Avg generation throughput: 3932.4 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 36.3% | |
| progress {"time": "11:38:30", "elapsed_min": 87.0, "submitted": 1843, "finished": 1587, "in_flight": 256, "gen_tokens": 19211146, "tok_per_s": 3680, "chain_tokens_median": 1777.5, "chain_tokens_p90": 6278.900000000001, "chain_tokens_max": 32674, "parse_rate": 0.9903906742281033, "looped": 61, "minutes_to_deadline": 91.4} | |
| (MainProcess pid=64676) INFO 10-31 11:38:30 [loggers.py:335] Engine 000: Avg prompt throughput: 328.6 tokens/s, Avg generation throughput: 3868.3 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 80.2%, Prefix cache hit rate: 36.8% | |
| progress {"time": "11:39:30", "elapsed_min": 88.0, "submitted": 1861, "finished": 1605, "in_flight": 256, "gen_tokens": 19441075, "tok_per_s": 3682, "chain_tokens_median": 1775.0, "chain_tokens_p90": 6282.1, "chain_tokens_max": 32674, "parse_rate": 0.9903426791277259, "looped": 62, "minutes_to_deadline": 90.4} | |
| (MainProcess pid=64676) INFO 10-31 11:39:30 [loggers.py:335] Engine 000: Avg prompt throughput: 310.5 tokens/s, Avg generation throughput: 3885.4 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 80.4%, Prefix cache hit rate: 36.7% | |
| progress {"time": "11:40:30", "elapsed_min": 89.0, "submitted": 1879, "finished": 1623, "in_flight": 256, "gen_tokens": 19684394, "tok_per_s": 3686, "chain_tokens_median": 1788.0, "chain_tokens_p90": 6280.700000000002, "chain_tokens_max": 32674, "parse_rate": 0.9902957486136784, "looped": 63, "minutes_to_deadline": 89.4} | |
| (MainProcess pid=64676) INFO 10-31 11:40:30 [loggers.py:335] Engine 000: Avg prompt throughput: 362.0 tokens/s, Avg generation throughput: 3821.1 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 80.0%, Prefix cache hit rate: 38.2% | |
| progress {"time": "11:41:30", "elapsed_min": 90.0, "submitted": 1897, "finished": 1641, "in_flight": 256, "gen_tokens": 19922449, "tok_per_s": 3689, "chain_tokens_median": 1799.0, "chain_tokens_p90": 6280.099999999999, "chain_tokens_max": 32674, "parse_rate": 0.9902498476538696, "looped": 64, "minutes_to_deadline": 88.4} | |
| (MainProcess pid=64676) INFO 10-31 11:41:30 [loggers.py:335] Engine 000: Avg prompt throughput: 388.8 tokens/s, Avg generation throughput: 3859.7 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 80.5%, Prefix cache hit rate: 39.3% | |
| progress {"time": "11:42:30", "elapsed_min": 91.0, "submitted": 1920, "finished": 1664, "in_flight": 256, "gen_tokens": 20268501, "tok_per_s": 3712, "chain_tokens_median": 1806.0, "chain_tokens_p90": 6281.5, "chain_tokens_max": 32674, "parse_rate": 0.9900841346153846, "looped": 66, "minutes_to_deadline": 87.4} | |
| (MainProcess pid=64676) INFO 10-31 11:42:30 [loggers.py:335] Engine 000: Avg prompt throughput: 346.3 tokens/s, Avg generation throughput: 3855.4 tokens/s, Running: 256 reqs, Waiting: 648 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 38.2% | |
| progress {"time": "11:43:30", "elapsed_min": 92.0, "submitted": 1937, "finished": 1681, "in_flight": 256, "gen_tokens": 20516258, "tok_per_s": 3716, "chain_tokens_median": 1804.5, "chain_tokens_p90": 6278.0, "chain_tokens_max": 32674, "parse_rate": 0.989738251041047, "looped": 69, "minutes_to_deadline": 86.4} | |
| (MainProcess pid=64676) INFO 10-31 11:43:30 [loggers.py:335] Engine 000: Avg prompt throughput: 397.0 tokens/s, Avg generation throughput: 3859.4 tokens/s, Running: 256 reqs, Waiting: 646 reqs, GPU KV cache usage: 76.3%, Prefix cache hit rate: 39.1% | |
| progress {"time": "11:44:30", "elapsed_min": 93.0, "submitted": 1953, "finished": 1697, "in_flight": 256, "gen_tokens": 20692848, "tok_per_s": 3708, "chain_tokens_median": 1802.0, "chain_tokens_p90": 6264.6, "chain_tokens_max": 32674, "parse_rate": 0.9896876841484974, "looped": 70, "minutes_to_deadline": 85.4} | |
| (MainProcess pid=64676) INFO 10-31 11:44:30 [loggers.py:335] Engine 000: Avg prompt throughput: 291.5 tokens/s, Avg generation throughput: 3919.5 tokens/s, Running: 255 reqs, Waiting: 642 reqs, GPU KV cache usage: 78.1%, Prefix cache hit rate: 38.5% | |
| progress {"time": "11:45:30", "elapsed_min": 94.0, "submitted": 1970, "finished": 1714, "in_flight": 256, "gen_tokens": 20871918, "tok_per_s": 3700, "chain_tokens_median": 1783.5, "chain_tokens_p90": 6250.5, "chain_tokens_max": 32674, "parse_rate": 0.9894982497082847, "looped": 72, "minutes_to_deadline": 84.4} | |
| (MainProcess pid=64676) INFO 10-31 11:45:30 [loggers.py:335] Engine 000: Avg prompt throughput: 342.1 tokens/s, Avg generation throughput: 3897.9 tokens/s, Running: 256 reqs, Waiting: 633 reqs, GPU KV cache usage: 77.6%, Prefix cache hit rate: 38.4% | |
| progress {"time": "11:46:30", "elapsed_min": 95.0, "submitted": 1982, "finished": 1726, "in_flight": 256, "gen_tokens": 21014148, "tok_per_s": 3686, "chain_tokens_median": 1789.5, "chain_tokens_p90": 6243.799999999999, "chain_tokens_max": 32674, "parse_rate": 0.9895712630359212, "looped": 72, "minutes_to_deadline": 83.4} | |
| (MainProcess pid=64676) INFO 10-31 11:46:30 [loggers.py:335] Engine 000: Avg prompt throughput: 250.5 tokens/s, Avg generation throughput: 3970.9 tokens/s, Running: 256 reqs, Waiting: 619 reqs, GPU KV cache usage: 80.2%, Prefix cache hit rate: 38.0% | |
| progress {"time": "11:47:30", "elapsed_min": 96.0, "submitted": 2001, "finished": 1745, "in_flight": 256, "gen_tokens": 21196727, "tok_per_s": 3680, "chain_tokens_median": 1778.5, "chain_tokens_p90": 6241.4000000000015, "chain_tokens_max": 32674, "parse_rate": 0.9896848137535816, "looped": 72, "minutes_to_deadline": 82.4} | |
| (MainProcess pid=64676) INFO 10-31 11:47:30 [loggers.py:335] Engine 000: Avg prompt throughput: 425.8 tokens/s, Avg generation throughput: 3786.9 tokens/s, Running: 256 reqs, Waiting: 612 reqs, GPU KV cache usage: 79.3%, Prefix cache hit rate: 37.5% | |
| progress {"time": "11:48:30", "elapsed_min": 97.0, "submitted": 2012, "finished": 1756, "in_flight": 256, "gen_tokens": 21358628, "tok_per_s": 3670, "chain_tokens_median": 1783.5, "chain_tokens_p90": 6243.799999999999, "chain_tokens_max": 32674, "parse_rate": 0.9896070615034168, "looped": 73, "minutes_to_deadline": 81.4} | |
| (MainProcess pid=64676) INFO 10-31 11:48:30 [loggers.py:335] Engine 000: Avg prompt throughput: 240.1 tokens/s, Avg generation throughput: 3889.8 tokens/s, Running: 256 reqs, Waiting: 594 reqs, GPU KV cache usage: 81.2%, Prefix cache hit rate: 37.3% | |
| progress {"time": "11:49:30", "elapsed_min": 98.0, "submitted": 2037, "finished": 1781, "in_flight": 256, "gen_tokens": 21677018, "tok_per_s": 3686, "chain_tokens_median": 1791.5, "chain_tokens_p90": 6251.899999999999, "chain_tokens_max": 32674, "parse_rate": 0.9896125772038181, "looped": 74, "minutes_to_deadline": 80.4} | |
| (MainProcess pid=64676) INFO 10-31 11:49:30 [loggers.py:335] Engine 000: Avg prompt throughput: 424.6 tokens/s, Avg generation throughput: 3765.6 tokens/s, Running: 256 reqs, Waiting: 612 reqs, GPU KV cache usage: 79.3%, Prefix cache hit rate: 37.0% | |
| progress {"time": "11:50:30", "elapsed_min": 99.0, "submitted": 2058, "finished": 1802, "in_flight": 256, "gen_tokens": 21944791, "tok_per_s": 3694, "chain_tokens_median": 1792.5, "chain_tokens_p90": 6255.800000000001, "chain_tokens_max": 32674, "parse_rate": 0.9895948945615982, "looped": 75, "minutes_to_deadline": 79.4} | |
| (MainProcess pid=64676) INFO 10-31 11:50:30 [loggers.py:335] Engine 000: Avg prompt throughput: 340.1 tokens/s, Avg generation throughput: 3850.9 tokens/s, Running: 256 reqs, Waiting: 615 reqs, GPU KV cache usage: 74.9%, Prefix cache hit rate: 36.3% | |
| progress {"time": "11:51:30", "elapsed_min": 100.0, "submitted": 2077, "finished": 1821, "in_flight": 256, "gen_tokens": 22217496, "tok_per_s": 3703, "chain_tokens_median": 1791.5, "chain_tokens_p90": 6267.4, "chain_tokens_max": 32674, "parse_rate": 0.9894288852278967, "looped": 77, "minutes_to_deadline": 78.4} | |
| (MainProcess pid=64676) INFO 10-31 11:51:30 [loggers.py:335] Engine 000: Avg prompt throughput: 341.3 tokens/s, Avg generation throughput: 4013.2 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 73.2%, Prefix cache hit rate: 37.8% | |
| progress {"time": "11:52:30", "elapsed_min": 101.0, "submitted": 2097, "finished": 1841, "in_flight": 256, "gen_tokens": 22558736, "tok_per_s": 3722, "chain_tokens_median": 1806.0, "chain_tokens_p90": 6298.7, "chain_tokens_max": 32674, "parse_rate": 0.9894079304725693, "looped": 78, "minutes_to_deadline": 77.4} | |
| (MainProcess pid=64676) INFO 10-31 11:52:30 [loggers.py:335] Engine 000: Avg prompt throughput: 341.6 tokens/s, Avg generation throughput: 4085.8 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 72.7%, Prefix cache hit rate: 37.6% | |
| progress {"time": "11:53:30", "elapsed_min": 102.0, "submitted": 2109, "finished": 1853, "in_flight": 256, "gen_tokens": 22721388, "tok_per_s": 3712, "chain_tokens_median": 1814.0, "chain_tokens_p90": 6310.700000000002, "chain_tokens_max": 32674, "parse_rate": 0.9894765245547761, "looped": 78, "minutes_to_deadline": 76.4} | |
| (MainProcess pid=64676) INFO 10-31 11:53:30 [loggers.py:335] Engine 000: Avg prompt throughput: 278.7 tokens/s, Avg generation throughput: 4124.4 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 76.3%, Prefix cache hit rate: 37.9% | |
| progress {"time": "11:54:30", "elapsed_min": 103.0, "submitted": 2124, "finished": 1868, "in_flight": 256, "gen_tokens": 22845109, "tok_per_s": 3696, "chain_tokens_median": 1808.5, "chain_tokens_p90": 6297.100000000005, "chain_tokens_max": 32674, "parse_rate": 0.9895610278372591, "looped": 78, "minutes_to_deadline": 75.4} | |
| (MainProcess pid=64676) INFO 10-31 11:54:30 [loggers.py:335] Engine 000: Avg prompt throughput: 383.9 tokens/s, Avg generation throughput: 3897.9 tokens/s, Running: 256 reqs, Waiting: 611 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 38.2% | |
| progress {"time": "11:55:30", "elapsed_min": 104.0, "submitted": 2148, "finished": 1892, "in_flight": 256, "gen_tokens": 23148589, "tok_per_s": 3709, "chain_tokens_median": 1802.0, "chain_tokens_p90": 6281.3, "chain_tokens_max": 32674, "parse_rate": 0.9894291754756871, "looped": 80, "minutes_to_deadline": 74.4} | |
| (MainProcess pid=64676) INFO 10-31 11:55:30 [loggers.py:335] Engine 000: Avg prompt throughput: 369.7 tokens/s, Avg generation throughput: 3833.9 tokens/s, Running: 255 reqs, Waiting: 622 reqs, GPU KV cache usage: 76.7%, Prefix cache hit rate: 37.3% | |
| progress {"time": "11:56:30", "elapsed_min": 105.0, "submitted": 2170, "finished": 1914, "in_flight": 256, "gen_tokens": 23407549, "tok_per_s": 3715, "chain_tokens_median": 1797.5, "chain_tokens_p90": 6279.5, "chain_tokens_max": 32674, "parse_rate": 0.9894200626959248, "looped": 81, "minutes_to_deadline": 73.4} | |
| (MainProcess pid=64676) INFO 10-31 11:56:30 [loggers.py:335] Engine 000: Avg prompt throughput: 381.7 tokens/s, Avg generation throughput: 3949.2 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 35.9% | |
| progress {"time": "11:57:30", "elapsed_min": 106.0, "submitted": 2180, "finished": 1924, "in_flight": 256, "gen_tokens": 23486719, "tok_per_s": 3693, "chain_tokens_median": 1788.0, "chain_tokens_p90": 6274.0, "chain_tokens_max": 32674, "parse_rate": 0.9894750519750519, "looped": 81, "minutes_to_deadline": 72.4} | |
| (MainProcess pid=64676) INFO 10-31 11:57:30 [loggers.py:335] Engine 000: Avg prompt throughput: 238.1 tokens/s, Avg generation throughput: 3962.4 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 81.3%, Prefix cache hit rate: 34.9% | |
| progress {"time": "11:58:30", "elapsed_min": 107.0, "submitted": 2199, "finished": 1943, "in_flight": 256, "gen_tokens": 23868786, "tok_per_s": 3718, "chain_tokens_median": 1816.0, "chain_tokens_p90": 6298.900000000001, "chain_tokens_max": 32674, "parse_rate": 0.9893206381883685, "looped": 83, "minutes_to_deadline": 71.4} | |
| (MainProcess pid=64676) INFO 10-31 11:58:30 [loggers.py:335] Engine 000: Avg prompt throughput: 287.9 tokens/s, Avg generation throughput: 3906.8 tokens/s, Running: 256 reqs, Waiting: 632 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 33.8% | |
| progress {"time": "11:59:30", "elapsed_min": 108.0, "submitted": 2216, "finished": 1960, "in_flight": 256, "gen_tokens": 24087549, "tok_per_s": 3717, "chain_tokens_median": 1816.0, "chain_tokens_p90": 6314.1, "chain_tokens_max": 32674, "parse_rate": 0.9894132653061225, "looped": 83, "minutes_to_deadline": 70.4} | |
| (MainProcess pid=64676) INFO 10-31 11:59:30 [loggers.py:335] Engine 000: Avg prompt throughput: 354.9 tokens/s, Avg generation throughput: 3910.7 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 79.6%, Prefix cache hit rate: 33.7% | |
| progress {"time": "12:00:30", "elapsed_min": 109.0, "submitted": 2233, "finished": 1977, "in_flight": 256, "gen_tokens": 24234447, "tok_per_s": 3705, "chain_tokens_median": 1803.0, "chain_tokens_p90": 6305.900000000001, "chain_tokens_max": 32674, "parse_rate": 0.9895042994436014, "looped": 83, "minutes_to_deadline": 69.4} | |
| (MainProcess pid=64676) INFO 10-31 12:00:30 [loggers.py:335] Engine 000: Avg prompt throughput: 342.6 tokens/s, Avg generation throughput: 3881.0 tokens/s, Running: 255 reqs, Waiting: 619 reqs, GPU KV cache usage: 81.4%, Prefix cache hit rate: 34.9% | |
| progress {"time": "12:01:30", "elapsed_min": 110.0, "submitted": 2253, "finished": 1997, "in_flight": 256, "gen_tokens": 24451618, "tok_per_s": 3705, "chain_tokens_median": 1800.5, "chain_tokens_p90": 6305.900000000001, "chain_tokens_max": 32674, "parse_rate": 0.9896094141211818, "looped": 83, "minutes_to_deadline": 68.4} | |
| (MainProcess pid=64676) INFO 10-31 12:01:30 [loggers.py:335] Engine 000: Avg prompt throughput: 375.4 tokens/s, Avg generation throughput: 3765.7 tokens/s, Running: 256 reqs, Waiting: 623 reqs, GPU KV cache usage: 83.0%, Prefix cache hit rate: 35.2% | |
| progress {"time": "12:02:30", "elapsed_min": 111.0, "submitted": 2266, "finished": 2010, "in_flight": 256, "gen_tokens": 24600474, "tok_per_s": 3694, "chain_tokens_median": 1800.5, "chain_tokens_p90": 6311.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9896766169154229, "looped": 83, "minutes_to_deadline": 67.4} | |
| (MainProcess pid=64676) INFO 10-31 12:02:30 [loggers.py:335] Engine 000: Avg prompt throughput: 327.1 tokens/s, Avg generation throughput: 3821.4 tokens/s, Running: 256 reqs, Waiting: 618 reqs, GPU KV cache usage: 85.3%, Prefix cache hit rate: 35.9% | |
| progress {"time": "12:03:30", "elapsed_min": 112.0, "submitted": 2283, "finished": 2027, "in_flight": 256, "gen_tokens": 24893755, "tok_per_s": 3704, "chain_tokens_median": 1814.0, "chain_tokens_p90": 6315.3, "chain_tokens_max": 32674, "parse_rate": 0.9895165268870252, "looped": 85, "minutes_to_deadline": 66.4} | |
| (MainProcess pid=64676) INFO 10-31 12:03:30 [loggers.py:335] Engine 000: Avg prompt throughput: 310.4 tokens/s, Avg generation throughput: 3787.3 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 84.4%, Prefix cache hit rate: 36.5% | |
| progress {"time": "12:04:30", "elapsed_min": 113.0, "submitted": 2296, "finished": 2040, "in_flight": 256, "gen_tokens": 25030170, "tok_per_s": 3692, "chain_tokens_median": 1814.0, "chain_tokens_p90": 6308.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9895833333333334, "looped": 85, "minutes_to_deadline": 65.4} | |
| (MainProcess pid=64676) INFO 10-31 12:04:30 [loggers.py:335] Engine 000: Avg prompt throughput: 284.9 tokens/s, Avg generation throughput: 3778.9 tokens/s, Running: 255 reqs, Waiting: 614 reqs, GPU KV cache usage: 87.8%, Prefix cache hit rate: 38.0% | |
| progress {"time": "12:05:30", "elapsed_min": 114.0, "submitted": 2310, "finished": 2054, "in_flight": 256, "gen_tokens": 25192762, "tok_per_s": 3683, "chain_tokens_median": 1814.0, "chain_tokens_p90": 6306.5, "chain_tokens_max": 32674, "parse_rate": 0.9896543330087634, "looped": 85, "minutes_to_deadline": 64.4} | |
| (MainProcess pid=64676) INFO 10-31 12:05:30 [loggers.py:335] Engine 000: Avg prompt throughput: 298.0 tokens/s, Avg generation throughput: 3723.4 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 88.6%, Prefix cache hit rate: 39.1% | |
| progress {"time": "12:06:30", "elapsed_min": 115.0, "submitted": 2326, "finished": 2070, "in_flight": 256, "gen_tokens": 25474442, "tok_per_s": 3692, "chain_tokens_median": 1821.5, "chain_tokens_p90": 6317.600000000002, "chain_tokens_max": 32674, "parse_rate": 0.9894927536231884, "looped": 87, "minutes_to_deadline": 63.4} | |
| (MainProcess pid=64676) INFO 10-31 12:06:30 [loggers.py:335] Engine 000: Avg prompt throughput: 297.9 tokens/s, Avg generation throughput: 3663.4 tokens/s, Running: 256 reqs, Waiting: 606 reqs, GPU KV cache usage: 85.4%, Prefix cache hit rate: 38.1% | |
| progress {"time": "12:07:30", "elapsed_min": 116.0, "submitted": 2344, "finished": 2088, "in_flight": 256, "gen_tokens": 25705542, "tok_per_s": 3693, "chain_tokens_median": 1825.0, "chain_tokens_p90": 6317.0, "chain_tokens_max": 32674, "parse_rate": 0.9894636015325671, "looped": 88, "minutes_to_deadline": 62.4} | |
| (MainProcess pid=64676) INFO 10-31 12:07:30 [loggers.py:335] Engine 000: Avg prompt throughput: 264.2 tokens/s, Avg generation throughput: 3761.8 tokens/s, Running: 256 reqs, Waiting: 615 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 37.0% | |
| progress {"time": "12:08:30", "elapsed_min": 117.0, "submitted": 2363, "finished": 2107, "in_flight": 256, "gen_tokens": 26129468, "tok_per_s": 3722, "chain_tokens_median": 1836.5, "chain_tokens_p90": 6347.500000000001, "chain_tokens_max": 32674, "parse_rate": 0.989202657807309, "looped": 91, "minutes_to_deadline": 61.4} | |
| (MainProcess pid=64676) INFO 10-31 12:08:30 [loggers.py:335] Engine 000: Avg prompt throughput: 317.4 tokens/s, Avg generation throughput: 3821.3 tokens/s, Running: 256 reqs, Waiting: 619 reqs, GPU KV cache usage: 80.6%, Prefix cache hit rate: 36.3% | |
| progress {"time": "12:09:30", "elapsed_min": 118.0, "submitted": 2383, "finished": 2127, "in_flight": 256, "gen_tokens": 26352383, "tok_per_s": 3722, "chain_tokens_median": 1835.5, "chain_tokens_p90": 6340.3, "chain_tokens_max": 32674, "parse_rate": 0.9893041842971321, "looped": 91, "minutes_to_deadline": 60.4} | |
| (MainProcess pid=64676) INFO 10-31 12:09:30 [loggers.py:335] Engine 000: Avg prompt throughput: 399.0 tokens/s, Avg generation throughput: 3829.6 tokens/s, Running: 256 reqs, Waiting: 619 reqs, GPU KV cache usage: 82.5%, Prefix cache hit rate: 36.7% | |
| progress {"time": "12:10:30", "elapsed_min": 119.0, "submitted": 2402, "finished": 2146, "in_flight": 256, "gen_tokens": 26507945, "tok_per_s": 3712, "chain_tokens_median": 1825.5, "chain_tokens_p90": 6338.199999999999, "chain_tokens_max": 32674, "parse_rate": 0.9893988816402609, "looped": 91, "minutes_to_deadline": 59.4} | |
| (MainProcess pid=64676) INFO 10-31 12:10:30 [loggers.py:335] Engine 000: Avg prompt throughput: 342.0 tokens/s, Avg generation throughput: 3804.2 tokens/s, Running: 256 reqs, Waiting: 627 reqs, GPU KV cache usage: 84.2%, Prefix cache hit rate: 36.1% | |
| progress {"time": "12:11:30", "elapsed_min": 120.0, "submitted": 2416, "finished": 2160, "in_flight": 256, "gen_tokens": 26644878, "tok_per_s": 3700, "chain_tokens_median": 1823.5, "chain_tokens_p90": 6323.700000000003, "chain_tokens_max": 32674, "parse_rate": 0.9894675925925925, "looped": 91, "minutes_to_deadline": 58.4} | |
| (MainProcess pid=64676) INFO 10-31 12:11:30 [loggers.py:335] Engine 000: Avg prompt throughput: 220.7 tokens/s, Avg generation throughput: 3796.0 tokens/s, Running: 256 reqs, Waiting: 626 reqs, GPU KV cache usage: 86.3%, Prefix cache hit rate: 36.7% | |
| progress {"time": "12:12:30", "elapsed_min": 121.0, "submitted": 2435, "finished": 2179, "in_flight": 256, "gen_tokens": 26951664, "tok_per_s": 3712, "chain_tokens_median": 1828.0, "chain_tokens_p90": 6340.5, "chain_tokens_max": 32674, "parse_rate": 0.9893299678751721, "looped": 93, "minutes_to_deadline": 57.4} | |
| (MainProcess pid=64676) INFO 10-31 12:12:30 [loggers.py:335] Engine 000: Avg prompt throughput: 284.7 tokens/s, Avg generation throughput: 3731.8 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 37.7% | |
| progress {"time": "12:13:30", "elapsed_min": 122.0, "submitted": 2452, "finished": 2196, "in_flight": 256, "gen_tokens": 27206794, "tok_per_s": 3717, "chain_tokens_median": 1825.0, "chain_tokens_p90": 6344.499999999999, "chain_tokens_max": 32674, "parse_rate": 0.9890710382513661, "looped": 96, "minutes_to_deadline": 56.4} | |
| (MainProcess pid=64676) INFO 10-31 12:13:30 [loggers.py:335] Engine 000: Avg prompt throughput: 270.3 tokens/s, Avg generation throughput: 3723.3 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 83.8%, Prefix cache hit rate: 39.1% | |
| progress {"time": "12:14:30", "elapsed_min": 123.0, "submitted": 2468, "finished": 2212, "in_flight": 256, "gen_tokens": 27400553, "tok_per_s": 3713, "chain_tokens_median": 1832.0, "chain_tokens_p90": 6340.3, "chain_tokens_max": 32674, "parse_rate": 0.9891500904159132, "looped": 96, "minutes_to_deadline": 55.4} | |
| (MainProcess pid=64676) INFO 10-31 12:14:30 [loggers.py:335] Engine 000: Avg prompt throughput: 425.8 tokens/s, Avg generation throughput: 3705.9 tokens/s, Running: 256 reqs, Waiting: 620 reqs, GPU KV cache usage: 83.3%, Prefix cache hit rate: 38.3% | |
| progress {"time": "12:15:30", "elapsed_min": 124.0, "submitted": 2492, "finished": 2236, "in_flight": 256, "gen_tokens": 27707717, "tok_per_s": 3724, "chain_tokens_median": 1824.5, "chain_tokens_p90": 6340.7, "chain_tokens_max": 32674, "parse_rate": 0.9891547406082289, "looped": 97, "minutes_to_deadline": 54.4} | |
| (MainProcess pid=64676) INFO 10-31 12:15:30 [loggers.py:335] Engine 000: Avg prompt throughput: 420.8 tokens/s, Avg generation throughput: 3718.3 tokens/s, Running: 256 reqs, Waiting: 619 reqs, GPU KV cache usage: 79.8%, Prefix cache hit rate: 37.3% | |
| progress {"time": "12:16:30", "elapsed_min": 125.0, "submitted": 2515, "finished": 2259, "in_flight": 256, "gen_tokens": 27901107, "tok_per_s": 3720, "chain_tokens_median": 1814.0, "chain_tokens_p90": 6331.5, "chain_tokens_max": 32674, "parse_rate": 0.9892651615759186, "looped": 97, "minutes_to_deadline": 53.4} | |
| (MainProcess pid=64676) INFO 10-31 12:16:30 [loggers.py:335] Engine 000: Avg prompt throughput: 440.2 tokens/s, Avg generation throughput: 3786.5 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 81.7%, Prefix cache hit rate: 37.6% | |
| progress {"time": "12:17:30", "elapsed_min": 126.0, "submitted": 2531, "finished": 2275, "in_flight": 256, "gen_tokens": 28026372, "tok_per_s": 3707, "chain_tokens_median": 1800.5, "chain_tokens_p90": 6317.0, "chain_tokens_max": 32674, "parse_rate": 0.9893406593406593, "looped": 97, "minutes_to_deadline": 52.4} | |
| (MainProcess pid=64676) INFO 10-31 12:17:30 [loggers.py:335] Engine 000: Avg prompt throughput: 340.4 tokens/s, Avg generation throughput: 3821.3 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 83.6%, Prefix cache hit rate: 36.8% | |
| progress {"time": "12:18:30", "elapsed_min": 127.0, "submitted": 2550, "finished": 2294, "in_flight": 256, "gen_tokens": 28308245, "tok_per_s": 3715, "chain_tokens_median": 1809.5, "chain_tokens_p90": 6340.0, "chain_tokens_max": 32674, "parse_rate": 0.9894289450741064, "looped": 97, "minutes_to_deadline": 51.4} | |
| (MainProcess pid=64676) INFO 10-31 12:18:30 [loggers.py:335] Engine 000: Avg prompt throughput: 398.9 tokens/s, Avg generation throughput: 3692.9 tokens/s, Running: 256 reqs, Waiting: 616 reqs, GPU KV cache usage: 84.1%, Prefix cache hit rate: 37.4% | |
| progress {"time": "12:19:30", "elapsed_min": 128.0, "submitted": 2565, "finished": 2309, "in_flight": 256, "gen_tokens": 28554592, "tok_per_s": 3718, "chain_tokens_median": 1820.0, "chain_tokens_p90": 6340.0, "chain_tokens_max": 32674, "parse_rate": 0.9892810740580338, "looped": 99, "minutes_to_deadline": 50.4} | |
| (MainProcess pid=64676) INFO 10-31 12:19:30 [loggers.py:335] Engine 000: Avg prompt throughput: 331.2 tokens/s, Avg generation throughput: 3851.4 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 85.2%, Prefix cache hit rate: 37.2% | |
| progress {"time": "12:20:30", "elapsed_min": 129.0, "submitted": 2581, "finished": 2325, "in_flight": 256, "gen_tokens": 28854377, "tok_per_s": 3728, "chain_tokens_median": 1826.0, "chain_tokens_p90": 6341.500000000002, "chain_tokens_max": 32674, "parse_rate": 0.9890322580645161, "looped": 102, "minutes_to_deadline": 49.4} | |
| (MainProcess pid=64676) INFO 10-31 12:20:30 [loggers.py:335] Engine 000: Avg prompt throughput: 295.1 tokens/s, Avg generation throughput: 3761.7 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 81.9%, Prefix cache hit rate: 36.9% | |
| progress {"time": "12:21:30", "elapsed_min": 130.0, "submitted": 2605, "finished": 2349, "in_flight": 256, "gen_tokens": 29124413, "tok_per_s": 3734, "chain_tokens_median": 1835.0, "chain_tokens_p90": 6340.0, "chain_tokens_max": 32674, "parse_rate": 0.9891443167305236, "looped": 102, "minutes_to_deadline": 48.4} | |
| (MainProcess pid=64676) INFO 10-31 12:21:30 [loggers.py:335] Engine 000: Avg prompt throughput: 363.0 tokens/s, Avg generation throughput: 3847.0 tokens/s, Running: 256 reqs, Waiting: 649 reqs, GPU KV cache usage: 82.3%, Prefix cache hit rate: 36.8% | |
| progress {"time": "12:22:30", "elapsed_min": 131.0, "submitted": 2617, "finished": 2361, "in_flight": 256, "gen_tokens": 29313382, "tok_per_s": 3729, "chain_tokens_median": 1836.5, "chain_tokens_p90": 6333.700000000001, "chain_tokens_max": 32674, "parse_rate": 0.9889877170690385, "looped": 104, "minutes_to_deadline": 47.4} | |
| (MainProcess pid=64676) INFO 10-31 12:22:30 [loggers.py:335] Engine 000: Avg prompt throughput: 353.1 tokens/s, Avg generation throughput: 3868.3 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 82.7%, Prefix cache hit rate: 36.5% | |
| progress {"time": "12:23:30", "elapsed_min": 132.0, "submitted": 2634, "finished": 2378, "in_flight": 256, "gen_tokens": 29525592, "tok_per_s": 3728, "chain_tokens_median": 1830.5, "chain_tokens_p90": 6339.399999999998, "chain_tokens_max": 32674, "parse_rate": 0.9888561816652649, "looped": 106, "minutes_to_deadline": 46.4} | |
| (MainProcess pid=64676) INFO 10-31 12:23:30 [loggers.py:335] Engine 000: Avg prompt throughput: 436.5 tokens/s, Avg generation throughput: 3761.3 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 81.1%, Prefix cache hit rate: 36.2% | |
| progress {"time": "12:24:30", "elapsed_min": 133.0, "submitted": 2655, "finished": 2399, "in_flight": 256, "gen_tokens": 29863973, "tok_per_s": 3742, "chain_tokens_median": 1833.0, "chain_tokens_p90": 6340.5, "chain_tokens_max": 32674, "parse_rate": 0.9886411004585244, "looped": 109, "minutes_to_deadline": 45.4} | |
| (MainProcess pid=64676) INFO 10-31 12:24:30 [loggers.py:335] Engine 000: Avg prompt throughput: 387.5 tokens/s, Avg generation throughput: 3859.4 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 78.5%, Prefix cache hit rate: 37.6% | |
| progress {"time": "12:25:30", "elapsed_min": 134.0, "submitted": 2674, "finished": 2418, "in_flight": 256, "gen_tokens": 30078667, "tok_per_s": 3741, "chain_tokens_median": 1835.0, "chain_tokens_p90": 6342.799999999999, "chain_tokens_max": 32674, "parse_rate": 0.9887303556658396, "looped": 109, "minutes_to_deadline": 44.4} | |
| (MainProcess pid=64676) INFO 10-31 12:25:30 [loggers.py:335] Engine 000: Avg prompt throughput: 307.9 tokens/s, Avg generation throughput: 3867.9 tokens/s, Running: 256 reqs, Waiting: 629 reqs, GPU KV cache usage: 77.4%, Prefix cache hit rate: 37.3% | |
| progress {"time": "12:26:30", "elapsed_min": 135.0, "submitted": 2698, "finished": 2442, "in_flight": 256, "gen_tokens": 30441779, "tok_per_s": 3758, "chain_tokens_median": 1839.5, "chain_tokens_p90": 6347.5000000000055, "chain_tokens_max": 32674, "parse_rate": 0.9885339885339886, "looped": 112, "minutes_to_deadline": 43.4} | |
| (MainProcess pid=64676) INFO 10-31 12:26:30 [loggers.py:335] Engine 000: Avg prompt throughput: 440.1 tokens/s, Avg generation throughput: 3910.3 tokens/s, Running: 256 reqs, Waiting: 630 reqs, GPU KV cache usage: 71.0%, Prefix cache hit rate: 36.5% | |
| progress {"time": "12:27:30", "elapsed_min": 136.0, "submitted": 2721, "finished": 2465, "in_flight": 256, "gen_tokens": 30761578, "tok_per_s": 3770, "chain_tokens_median": 1843.5, "chain_tokens_p90": 6354.0, "chain_tokens_max": 32674, "parse_rate": 0.9884381338742394, "looped": 114, "minutes_to_deadline": 42.4} | |
| (MainProcess pid=64676) INFO 10-31 12:27:30 [loggers.py:335] Engine 000: Avg prompt throughput: 359.4 tokens/s, Avg generation throughput: 4141.2 tokens/s, Running: 255 reqs, Waiting: 648 reqs, GPU KV cache usage: 70.2%, Prefix cache hit rate: 36.2% | |
| progress {"time": "12:28:30", "elapsed_min": 137.0, "submitted": 2741, "finished": 2485, "in_flight": 256, "gen_tokens": 31047167, "tok_per_s": 3777, "chain_tokens_median": 1843.5, "chain_tokens_p90": 6358.1, "chain_tokens_max": 32674, "parse_rate": 0.9883299798792756, "looped": 116, "minutes_to_deadline": 41.4} | |
| (MainProcess pid=64676) INFO 10-31 12:28:30 [loggers.py:335] Engine 000: Avg prompt throughput: 357.8 tokens/s, Avg generation throughput: 4175.2 tokens/s, Running: 256 reqs, Waiting: 654 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 36.8% | |
| progress {"time": "12:29:30", "elapsed_min": 138.0, "submitted": 2758, "finished": 2502, "in_flight": 256, "gen_tokens": 31224877, "tok_per_s": 3771, "chain_tokens_median": 1840.5, "chain_tokens_p90": 6354.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9884092725819345, "looped": 116, "minutes_to_deadline": 40.4} | |
| (MainProcess pid=64676) INFO 10-31 12:29:30 [loggers.py:335] Engine 000: Avg prompt throughput: 421.9 tokens/s, Avg generation throughput: 4051.3 tokens/s, Running: 256 reqs, Waiting: 641 reqs, GPU KV cache usage: 72.1%, Prefix cache hit rate: 36.3% | |
| progress {"time": "12:30:30", "elapsed_min": 139.0, "submitted": 2776, "finished": 2520, "in_flight": 256, "gen_tokens": 31448411, "tok_per_s": 3771, "chain_tokens_median": 1847.5, "chain_tokens_p90": 6355.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9884920634920635, "looped": 116, "minutes_to_deadline": 39.4} | |
| (MainProcess pid=64676) INFO 10-31 12:30:30 [loggers.py:335] Engine 000: Avg prompt throughput: 404.3 tokens/s, Avg generation throughput: 4034.4 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 74.5%, Prefix cache hit rate: 38.0% | |
| progress {"time": "12:31:30", "elapsed_min": 140.0, "submitted": 2799, "finished": 2543, "in_flight": 256, "gen_tokens": 31750407, "tok_per_s": 3780, "chain_tokens_median": 1844.0, "chain_tokens_p90": 6354.9, "chain_tokens_max": 32674, "parse_rate": 0.988399528116398, "looped": 118, "minutes_to_deadline": 38.4} | |
| (MainProcess pid=64676) INFO 10-31 12:31:30 [loggers.py:335] Engine 000: Avg prompt throughput: 334.9 tokens/s, Avg generation throughput: 4004.2 tokens/s, Running: 256 reqs, Waiting: 659 reqs, GPU KV cache usage: 72.7%, Prefix cache hit rate: 37.4% | |
| progress {"time": "12:32:30", "elapsed_min": 141.0, "submitted": 2817, "finished": 2561, "in_flight": 256, "gen_tokens": 32023132, "tok_per_s": 3785, "chain_tokens_median": 1854.0, "chain_tokens_p90": 6359.0, "chain_tokens_max": 32674, "parse_rate": 0.9883834439672003, "looped": 119, "minutes_to_deadline": 37.4} | |
| (MainProcess pid=64676) INFO 10-31 12:32:30 [loggers.py:335] Engine 000: Avg prompt throughput: 410.6 tokens/s, Avg generation throughput: 4030.1 tokens/s, Running: 256 reqs, Waiting: 656 reqs, GPU KV cache usage: 72.9%, Prefix cache hit rate: 37.6% | |
| progress {"time": "12:33:30", "elapsed_min": 142.0, "submitted": 2835, "finished": 2579, "in_flight": 256, "gen_tokens": 32199087, "tok_per_s": 3779, "chain_tokens_median": 1855.5, "chain_tokens_p90": 6354.5, "chain_tokens_max": 32674, "parse_rate": 0.9884645211322218, "looped": 119, "minutes_to_deadline": 36.4} | |
| (MainProcess pid=64676) INFO 10-31 12:33:30 [loggers.py:335] Engine 000: Avg prompt throughput: 325.1 tokens/s, Avg generation throughput: 4004.5 tokens/s, Running: 256 reqs, Waiting: 653 reqs, GPU KV cache usage: 74.4%, Prefix cache hit rate: 37.0% | |
| progress {"time": "12:34:30", "elapsed_min": 143.0, "submitted": 2852, "finished": 2596, "in_flight": 256, "gen_tokens": 32476665, "tok_per_s": 3785, "chain_tokens_median": 1874.5, "chain_tokens_p90": 6363.200000000004, "chain_tokens_max": 32674, "parse_rate": 0.9884437596302003, "looped": 120, "minutes_to_deadline": 35.4} | |
| (MainProcess pid=64676) INFO 10-31 12:34:30 [loggers.py:335] Engine 000: Avg prompt throughput: 352.2 tokens/s, Avg generation throughput: 4081.4 tokens/s, Running: 256 reqs, Waiting: 660 reqs, GPU KV cache usage: 73.2%, Prefix cache hit rate: 38.9% | |
| progress {"time": "12:35:30", "elapsed_min": 144.0, "submitted": 2863, "finished": 2607, "in_flight": 256, "gen_tokens": 32575031, "tok_per_s": 3770, "chain_tokens_median": 1869.5, "chain_tokens_p90": 6359.0, "chain_tokens_max": 32674, "parse_rate": 0.9884925201380897, "looped": 120, "minutes_to_deadline": 34.4} | |
| (MainProcess pid=64676) INFO 10-31 12:35:31 [loggers.py:335] Engine 000: Avg prompt throughput: 339.2 tokens/s, Avg generation throughput: 4013.2 tokens/s, Running: 256 reqs, Waiting: 640 reqs, GPU KV cache usage: 75.6%, Prefix cache hit rate: 39.7% | |
| progress {"time": "12:36:31", "elapsed_min": 145.0, "submitted": 2879, "finished": 2623, "in_flight": 256, "gen_tokens": 32758397, "tok_per_s": 3765, "chain_tokens_median": 1865.0, "chain_tokens_p90": 6358.9, "chain_tokens_max": 32674, "parse_rate": 0.9884674037361799, "looped": 121, "minutes_to_deadline": 33.4} | |
| (MainProcess pid=64676) INFO 10-31 12:36:31 [loggers.py:335] Engine 000: Avg prompt throughput: 453.2 tokens/s, Avg generation throughput: 3867.8 tokens/s, Running: 256 reqs, Waiting: 618 reqs, GPU KV cache usage: 75.1%, Prefix cache hit rate: 40.5% | |
| progress {"time": "12:37:31", "elapsed_min": 146.0, "submitted": 2905, "finished": 2649, "in_flight": 256, "gen_tokens": 33134733, "tok_per_s": 3782, "chain_tokens_median": 1873.5, "chain_tokens_p90": 6380.0, "chain_tokens_max": 32674, "parse_rate": 0.9885805964514911, "looped": 121, "minutes_to_deadline": 32.4} | |
| (MainProcess pid=64676) INFO 10-31 12:37:31 [loggers.py:335] Engine 000: Avg prompt throughput: 422.0 tokens/s, Avg generation throughput: 3910.1 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 72.7%, Prefix cache hit rate: 39.5% | |
| progress {"time": "12:38:31", "elapsed_min": 147.0, "submitted": 2926, "finished": 2670, "in_flight": 256, "gen_tokens": 33444683, "tok_per_s": 3792, "chain_tokens_median": 1870.5, "chain_tokens_p90": 6394.1, "chain_tokens_max": 32674, "parse_rate": 0.9884831460674157, "looped": 123, "minutes_to_deadline": 31.4} | |
| (MainProcess pid=64676) INFO 10-31 12:38:31 [loggers.py:335] Engine 000: Avg prompt throughput: 362.3 tokens/s, Avg generation throughput: 4081.5 tokens/s, Running: 256 reqs, Waiting: 652 reqs, GPU KV cache usage: 71.5%, Prefix cache hit rate: 39.4% | |
| progress {"time": "12:39:31", "elapsed_min": 148.0, "submitted": 2941, "finished": 2685, "in_flight": 256, "gen_tokens": 33629143, "tok_per_s": 3787, "chain_tokens_median": 1874.5, "chain_tokens_p90": 6392.200000000001, "chain_tokens_max": 32674, "parse_rate": 0.9885474860335196, "looped": 123, "minutes_to_deadline": 30.4} | |
| (MainProcess pid=64676) INFO 10-31 12:39:31 [loggers.py:335] Engine 000: Avg prompt throughput: 524.5 tokens/s, Avg generation throughput: 4017.4 tokens/s, Running: 256 reqs, Waiting: 649 reqs, GPU KV cache usage: 75.6%, Prefix cache hit rate: 42.9% | |
| progress {"time": "12:40:31", "elapsed_min": 149.0, "submitted": 2955, "finished": 2699, "in_flight": 256, "gen_tokens": 33765959, "tok_per_s": 3777, "chain_tokens_median": 1870.0, "chain_tokens_p90": 6386.5, "chain_tokens_max": 32674, "parse_rate": 0.9886068914412746, "looped": 123, "minutes_to_deadline": 29.4} | |
| (MainProcess pid=64676) INFO 10-31 12:40:31 [loggers.py:335] Engine 000: Avg prompt throughput: 442.3 tokens/s, Avg generation throughput: 3910.7 tokens/s, Running: 256 reqs, Waiting: 631 reqs, GPU KV cache usage: 76.8%, Prefix cache hit rate: 44.3% | |
| progress {"time": "12:41:31", "elapsed_min": 150.0, "submitted": 2976, "finished": 2720, "in_flight": 256, "gen_tokens": 34003411, "tok_per_s": 3778, "chain_tokens_median": 1870.0, "chain_tokens_p90": 6390.1, "chain_tokens_max": 32674, "parse_rate": 0.9886948529411764, "looped": 123, "minutes_to_deadline": 28.4} | |
| (MainProcess pid=64676) INFO 10-31 12:41:31 [loggers.py:335] Engine 000: Avg prompt throughput: 575.1 tokens/s, Avg generation throughput: 3820.9 tokens/s, Running: 256 reqs, Waiting: 628 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 45.8% | |
| progress {"time": "12:42:31", "elapsed_min": 151.0, "submitted": 3000, "finished": 2744, "in_flight": 256, "gen_tokens": 34356475, "tok_per_s": 3792, "chain_tokens_median": 1877.5, "chain_tokens_p90": 6380.0, "chain_tokens_max": 32674, "parse_rate": 0.9885204081632653, "looped": 126, "minutes_to_deadline": 27.4} | |
| (MainProcess pid=64676) INFO 10-31 12:42:31 [loggers.py:335] Engine 000: Avg prompt throughput: 372.7 tokens/s, Avg generation throughput: 3944.8 tokens/s, Running: 256 reqs, Waiting: 645 reqs, GPU KV cache usage: 72.1%, Prefix cache hit rate: 45.6% | |
| progress {"time": "12:43:31", "elapsed_min": 152.0, "submitted": 3022, "finished": 2766, "in_flight": 256, "gen_tokens": 34733481, "tok_per_s": 3808, "chain_tokens_median": 1893.5, "chain_tokens_p90": 6390.700000000001, "chain_tokens_max": 32674, "parse_rate": 0.9885213304410702, "looped": 127, "minutes_to_deadline": 26.4} | |
| (MainProcess pid=64676) INFO 10-31 12:43:31 [loggers.py:335] Engine 000: Avg prompt throughput: 333.4 tokens/s, Avg generation throughput: 4145.5 tokens/s, Running: 256 reqs, Waiting: 660 reqs, GPU KV cache usage: 69.7%, Prefix cache hit rate: 45.3% | |
| progress {"time": "12:44:31", "elapsed_min": 153.0, "submitted": 3038, "finished": 2782, "in_flight": 256, "gen_tokens": 34928200, "tok_per_s": 3805, "chain_tokens_median": 1890.5, "chain_tokens_p90": 6394.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9885873472322071, "looped": 127, "minutes_to_deadline": 25.4} | |
| (MainProcess pid=64676) INFO 10-31 12:44:31 [loggers.py:335] Engine 000: Avg prompt throughput: 378.3 tokens/s, Avg generation throughput: 4115.4 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 70.9%, Prefix cache hit rate: 44.9% | |
| progress {"time": "12:45:31", "elapsed_min": 154.0, "submitted": 3060, "finished": 2804, "in_flight": 256, "gen_tokens": 35057037, "tok_per_s": 3794, "chain_tokens_median": 1870.0, "chain_tokens_p90": 6381.5, "chain_tokens_max": 32674, "parse_rate": 0.9886768901569187, "looped": 127, "minutes_to_deadline": 24.4} | |
| (MainProcess pid=64676) INFO 10-31 12:45:31 [loggers.py:335] Engine 000: Avg prompt throughput: 411.5 tokens/s, Avg generation throughput: 3957.2 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 73.8%, Prefix cache hit rate: 44.6% | |
| progress {"time": "12:46:31", "elapsed_min": 155.0, "submitted": 3080, "finished": 2824, "in_flight": 256, "gen_tokens": 35271121, "tok_per_s": 3792, "chain_tokens_median": 1870.0, "chain_tokens_p90": 6377.0, "chain_tokens_max": 32674, "parse_rate": 0.9887570821529745, "looped": 127, "minutes_to_deadline": 23.4} | |
| (MainProcess pid=64676) INFO 10-31 12:46:31 [loggers.py:335] Engine 000: Avg prompt throughput: 380.0 tokens/s, Avg generation throughput: 3949.0 tokens/s, Running: 256 reqs, Waiting: 637 reqs, GPU KV cache usage: 75.2%, Prefix cache hit rate: 44.5% | |
| progress {"time": "12:47:31", "elapsed_min": 156.0, "submitted": 3104, "finished": 2848, "in_flight": 256, "gen_tokens": 35601253, "tok_per_s": 3803, "chain_tokens_median": 1865.5, "chain_tokens_p90": 6375.699999999999, "chain_tokens_max": 32674, "parse_rate": 0.9885884831460674, "looped": 130, "minutes_to_deadline": 22.4} | |
| (MainProcess pid=64676) INFO 10-31 12:47:31 [loggers.py:335] Engine 000: Avg prompt throughput: 420.0 tokens/s, Avg generation throughput: 4004.5 tokens/s, Running: 256 reqs, Waiting: 655 reqs, GPU KV cache usage: 73.0%, Prefix cache hit rate: 42.8% | |
| progress {"time": "12:48:31", "elapsed_min": 157.0, "submitted": 3118, "finished": 2862, "in_flight": 256, "gen_tokens": 35780794, "tok_per_s": 3798, "chain_tokens_median": 1864.0, "chain_tokens_p90": 6382.0, "chain_tokens_max": 32674, "parse_rate": 0.9886443046820406, "looped": 130, "minutes_to_deadline": 21.4} | |
| (MainProcess pid=64676) INFO 10-31 12:48:31 [loggers.py:335] Engine 000: Avg prompt throughput: 337.9 tokens/s, Avg generation throughput: 4034.5 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 73.2%, Prefix cache hit rate: 42.5% | |
| progress {"time": "12:49:31", "elapsed_min": 158.0, "submitted": 3136, "finished": 2880, "in_flight": 256, "gen_tokens": 36000711, "tok_per_s": 3797, "chain_tokens_median": 1863.0, "chain_tokens_p90": 6383.700000000003, "chain_tokens_max": 32674, "parse_rate": 0.9886284722222223, "looped": 131, "minutes_to_deadline": 20.4} | |
| (MainProcess pid=64676) INFO 10-31 12:49:31 [loggers.py:335] Engine 000: Avg prompt throughput: 327.0 tokens/s, Avg generation throughput: 4004.5 tokens/s, Running: 256 reqs, Waiting: 633 reqs, GPU KV cache usage: 73.9%, Prefix cache hit rate: 42.3% | |
| progress {"time": "12:50:31", "elapsed_min": 159.0, "submitted": 3155, "finished": 2899, "in_flight": 256, "gen_tokens": 36241409, "tok_per_s": 3799, "chain_tokens_median": 1865.5, "chain_tokens_p90": 6391.5, "chain_tokens_max": 32674, "parse_rate": 0.9887030010348395, "looped": 131, "minutes_to_deadline": 19.4} | |
| (MainProcess pid=64676) INFO 10-31 12:50:31 [loggers.py:335] Engine 000: Avg prompt throughput: 385.6 tokens/s, Avg generation throughput: 4085.5 tokens/s, Running: 256 reqs, Waiting: 625 reqs, GPU KV cache usage: 74.8%, Prefix cache hit rate: 42.9% | |
| progress {"time": "12:51:31", "elapsed_min": 160.0, "submitted": 3177, "finished": 2921, "in_flight": 256, "gen_tokens": 36527838, "tok_per_s": 3805, "chain_tokens_median": 1870.5, "chain_tokens_p90": 6394.700000000001, "chain_tokens_max": 32674, "parse_rate": 0.9887880862718247, "looped": 131, "minutes_to_deadline": 18.4} | |
| (MainProcess pid=64676) INFO 10-31 12:51:31 [loggers.py:335] Engine 000: Avg prompt throughput: 322.0 tokens/s, Avg generation throughput: 4124.0 tokens/s, Running: 256 reqs, Waiting: 635 reqs, GPU KV cache usage: 75.1%, Prefix cache hit rate: 42.6% | |
| progress {"time": "12:52:31", "elapsed_min": 161.0, "submitted": 3192, "finished": 2936, "in_flight": 256, "gen_tokens": 36630673, "tok_per_s": 3792, "chain_tokens_median": 1858.0, "chain_tokens_p90": 6382.700000000001, "chain_tokens_max": 32674, "parse_rate": 0.9888453678474114, "looped": 131, "minutes_to_deadline": 17.4} | |
| (MainProcess pid=64676) INFO 10-31 12:52:31 [loggers.py:335] Engine 000: Avg prompt throughput: 338.4 tokens/s, Avg generation throughput: 4051.7 tokens/s, Running: 256 reqs, Waiting: 636 reqs, GPU KV cache usage: 79.1%, Prefix cache hit rate: 38.8% | |
| progress {"time": "12:53:31", "elapsed_min": 162.0, "submitted": 3203, "finished": 2947, "in_flight": 256, "gen_tokens": 36787374, "tok_per_s": 3784, "chain_tokens_median": 1865.0, "chain_tokens_p90": 6385.100000000008, "chain_tokens_max": 32674, "parse_rate": 0.9888870037326094, "looped": 131, "minutes_to_deadline": 16.4} | |
| (MainProcess pid=64676) INFO 10-31 12:53:31 [loggers.py:335] Engine 000: Avg prompt throughput: 214.0 tokens/s, Avg generation throughput: 4017.7 tokens/s, Running: 256 reqs, Waiting: 622 reqs, GPU KV cache usage: 81.3%, Prefix cache hit rate: 37.7% | |
| progress {"time": "12:54:31", "elapsed_min": 163.0, "submitted": 3218, "finished": 2962, "in_flight": 256, "gen_tokens": 36921586, "tok_per_s": 3775, "chain_tokens_median": 1862.0, "chain_tokens_p90": 6378.900000000003, "chain_tokens_max": 32674, "parse_rate": 0.9889432815665091, "looped": 131, "minutes_to_deadline": 15.4} | |
| (MainProcess pid=64676) INFO 10-31 12:54:31 [loggers.py:335] Engine 000: Avg prompt throughput: 341.6 tokens/s, Avg generation throughput: 3812.6 tokens/s, Running: 256 reqs, Waiting: 609 reqs, GPU KV cache usage: 82.3%, Prefix cache hit rate: 34.2% | |
| progress {"time": "12:55:31", "elapsed_min": 164.0, "submitted": 3240, "finished": 2984, "in_flight": 256, "gen_tokens": 37207920, "tok_per_s": 3781, "chain_tokens_median": 1868.0, "chain_tokens_p90": 6386.5, "chain_tokens_max": 32674, "parse_rate": 0.989024798927614, "looped": 131, "minutes_to_deadline": 14.4} | |
| (MainProcess pid=64676) INFO 10-31 12:55:31 [loggers.py:335] Engine 000: Avg prompt throughput: 399.6 tokens/s, Avg generation throughput: 3795.5 tokens/s, Running: 256 reqs, Waiting: 621 reqs, GPU KV cache usage: 83.0%, Prefix cache hit rate: 34.8% | |
| progress {"time": "12:56:31", "elapsed_min": 165.0, "submitted": 3259, "finished": 3003, "in_flight": 256, "gen_tokens": 37422294, "tok_per_s": 3780, "chain_tokens_median": 1870.0, "chain_tokens_p90": 6382.9, "chain_tokens_max": 32674, "parse_rate": 0.9890942390942391, "looped": 131, "minutes_to_deadline": 13.4} | |
| (MainProcess pid=64676) INFO 10-31 12:56:31 [loggers.py:335] Engine 000: Avg prompt throughput: 367.6 tokens/s, Avg generation throughput: 3757.1 tokens/s, Running: 256 reqs, Waiting: 617 reqs, GPU KV cache usage: 83.9%, Prefix cache hit rate: 35.9% | |
| progress {"time": "12:57:31", "elapsed_min": 166.0, "submitted": 3271, "finished": 3015, "in_flight": 256, "gen_tokens": 37520470, "tok_per_s": 3767, "chain_tokens_median": 1864.0, "chain_tokens_p90": 6381.1, "chain_tokens_max": 32674, "parse_rate": 0.9891376451077943, "looped": 131, "minutes_to_deadline": 12.4} | |
| (MainProcess pid=64676) INFO 10-31 12:57:31 [loggers.py:335] Engine 000: Avg prompt throughput: 297.9 tokens/s, Avg generation throughput: 3804.5 tokens/s, Running: 256 reqs, Waiting: 609 reqs, GPU KV cache usage: 85.8%, Prefix cache hit rate: 35.8% | |
| progress {"time": "12:58:31", "elapsed_min": 167.0, "submitted": 3287, "finished": 3031, "in_flight": 256, "gen_tokens": 37800150, "tok_per_s": 3772, "chain_tokens_median": 1869.5, "chain_tokens_p90": 6391.700000000001, "chain_tokens_max": 32674, "parse_rate": 0.9890300230946882, "looped": 133, "minutes_to_deadline": 11.4} | |
| (MainProcess pid=64676) INFO 10-31 12:58:31 [loggers.py:335] Engine 000: Avg prompt throughput: 493.9 tokens/s, Avg generation throughput: 3735.9 tokens/s, Running: 256 reqs, Waiting: 607 reqs, GPU KV cache usage: 85.4%, Prefix cache hit rate: 38.5% | |
| progress {"time": "12:59:31", "elapsed_min": 168.0, "submitted": 3305, "finished": 3049, "in_flight": 256, "gen_tokens": 38046393, "tok_per_s": 3774, "chain_tokens_median": 1870.0, "chain_tokens_p90": 6395.0, "chain_tokens_max": 32674, "parse_rate": 0.9890127910790423, "looped": 134, "minutes_to_deadline": 10.4} | |
| (MainProcess pid=64676) INFO 10-31 12:59:31 [loggers.py:335] Engine 000: Avg prompt throughput: 402.5 tokens/s, Avg generation throughput: 3663.2 tokens/s, Running: 256 reqs, Waiting: 598 reqs, GPU KV cache usage: 83.7%, Prefix cache hit rate: 37.8% | |
| progress {"time": "13:00:31", "elapsed_min": 169.0, "submitted": 3325, "finished": 3069, "in_flight": 256, "gen_tokens": 38336063, "tok_per_s": 3780, "chain_tokens_median": 1871.5, "chain_tokens_p90": 6401.5, "chain_tokens_max": 32674, "parse_rate": 0.9889214727924406, "looped": 136, "minutes_to_deadline": 9.4} | |
| (MainProcess pid=64676) INFO 10-31 13:00:31 [loggers.py:335] Engine 000: Avg prompt throughput: 317.1 tokens/s, Avg generation throughput: 3774.0 tokens/s, Running: 256 reqs, Waiting: 605 reqs, GPU KV cache usage: 81.2%, Prefix cache hit rate: 38.4% | |
| progress {"time": "13:01:31", "elapsed_min": 170.0, "submitted": 3352, "finished": 3096, "in_flight": 256, "gen_tokens": 38685212, "tok_per_s": 3792, "chain_tokens_median": 1868.0, "chain_tokens_p90": 6399.200000000004, "chain_tokens_max": 32674, "parse_rate": 0.9887758397932817, "looped": 139, "minutes_to_deadline": 8.4} | |
| (MainProcess pid=64676) INFO 10-31 13:01:31 [loggers.py:335] Engine 000: Avg prompt throughput: 533.2 tokens/s, Avg generation throughput: 3777.9 tokens/s, Running: 255 reqs, Waiting: 618 reqs, GPU KV cache usage: 76.6%, Prefix cache hit rate: 39.5% | |
| progress {"time": "13:02:31", "elapsed_min": 171.0, "submitted": 3383, "finished": 3127, "in_flight": 256, "gen_tokens": 39114629, "tok_per_s": 3812, "chain_tokens_median": 1875.0, "chain_tokens_p90": 6395.0, "chain_tokens_max": 32674, "parse_rate": 0.9887272145826671, "looped": 141, "minutes_to_deadline": 7.4} | |
| (MainProcess pid=64676) INFO 10-31 13:02:31 [loggers.py:335] Engine 000: Avg prompt throughput: 445.7 tokens/s, Avg generation throughput: 3876.0 tokens/s, Running: 255 reqs, Waiting: 647 reqs, GPU KV cache usage: 72.7%, Prefix cache hit rate: 39.6% | |
| progress {"time": "13:03:31", "elapsed_min": 172.0, "submitted": 3400, "finished": 3144, "in_flight": 256, "gen_tokens": 39265867, "tok_per_s": 3805, "chain_tokens_median": 1869.5, "chain_tokens_p90": 6390.5, "chain_tokens_max": 32674, "parse_rate": 0.9887881679389313, "looped": 141, "minutes_to_deadline": 6.4} | |
| (MainProcess pid=64676) INFO 10-31 13:03:31 [loggers.py:335] Engine 000: Avg prompt throughput: 317.5 tokens/s, Avg generation throughput: 4085.8 tokens/s, Running: 256 reqs, Waiting: 646 reqs, GPU KV cache usage: 76.0%, Prefix cache hit rate: 39.4% | |
| progress {"time": "13:04:31", "elapsed_min": 173.0, "submitted": 3414, "finished": 3158, "in_flight": 256, "gen_tokens": 39375299, "tok_per_s": 3793, "chain_tokens_median": 1866.5, "chain_tokens_p90": 6380.699999999999, "chain_tokens_max": 32674, "parse_rate": 0.988837872070931, "looped": 141, "minutes_to_deadline": 5.4} | |
| (MainProcess pid=64676) INFO 10-31 13:04:31 [loggers.py:335] Engine 000: Avg prompt throughput: 217.0 tokens/s, Avg generation throughput: 4021.9 tokens/s, Running: 256 reqs, Waiting: 642 reqs, GPU KV cache usage: 81.3%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:05:31", "elapsed_min": 174.0, "submitted": 3427, "finished": 3171, "in_flight": 256, "gen_tokens": 39532426, "tok_per_s": 3786, "chain_tokens_median": 1864.5, "chain_tokens_p90": 6381.700000000001, "chain_tokens_max": 32674, "parse_rate": 0.988804793440555, "looped": 142, "minutes_to_deadline": 4.4} | |
| (MainProcess pid=64676) INFO 10-31 13:05:31 [loggers.py:335] Engine 000: Avg prompt throughput: 411.6 tokens/s, Avg generation throughput: 3761.4 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 82.3%, Prefix cache hit rate: 41.0% | |
| progress {"time": "13:06:31", "elapsed_min": 175.0, "submitted": 3445, "finished": 3189, "in_flight": 256, "gen_tokens": 39827028, "tok_per_s": 3793, "chain_tokens_median": 1869.5, "chain_tokens_p90": 6382.5, "chain_tokens_max": 32674, "parse_rate": 0.9886328002508623, "looped": 145, "minutes_to_deadline": 3.4} | |
| (MainProcess pid=64676) INFO 10-31 13:06:31 [loggers.py:335] Engine 000: Avg prompt throughput: 384.2 tokens/s, Avg generation throughput: 3885.3 tokens/s, Running: 256 reqs, Waiting: 629 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 41.4% | |
| progress {"time": "13:07:31", "elapsed_min": 176.0, "submitted": 3466, "finished": 3210, "in_flight": 256, "gen_tokens": 40027986, "tok_per_s": 3790, "chain_tokens_median": 1865.5, "chain_tokens_p90": 6376.200000000001, "chain_tokens_max": 32674, "parse_rate": 0.9887071651090342, "looped": 145, "minutes_to_deadline": 2.4} | |
| (MainProcess pid=64676) INFO 10-31 13:07:31 [loggers.py:335] Engine 000: Avg prompt throughput: 345.6 tokens/s, Avg generation throughput: 3863.7 tokens/s, Running: 256 reqs, Waiting: 638 reqs, GPU KV cache usage: 81.6%, Prefix cache hit rate: 41.6% | |
| progress {"time": "13:08:31", "elapsed_min": 177.0, "submitted": 3484, "finished": 3228, "in_flight": 256, "gen_tokens": 40154160, "tok_per_s": 3781, "chain_tokens_median": 1854.5, "chain_tokens_p90": 6367.9, "chain_tokens_max": 32674, "parse_rate": 0.988770136307311, "looped": 145, "minutes_to_deadline": 1.4} | |
| (MainProcess pid=64676) INFO 10-31 13:08:31 [loggers.py:335] Engine 000: Avg prompt throughput: 487.4 tokens/s, Avg generation throughput: 3722.6 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 82.9%, Prefix cache hit rate: 43.5% | |
| progress {"time": "13:09:31", "elapsed_min": 178.0, "submitted": 3495, "finished": 3239, "in_flight": 256, "gen_tokens": 40282065, "tok_per_s": 3771, "chain_tokens_median": 1855.5, "chain_tokens_p90": 6368.5, "chain_tokens_max": 32674, "parse_rate": 0.988808274158691, "looped": 145, "minutes_to_deadline": 0.4} | |
| (MainProcess pid=64676) INFO 10-31 13:09:31 [loggers.py:335] Engine 000: Avg prompt throughput: 226.0 tokens/s, Avg generation throughput: 3949.7 tokens/s, Running: 256 reqs, Waiting: 624 reqs, GPU KV cache usage: 86.7%, Prefix cache hit rate: 44.1% | |
| progress {"time": "13:10:31", "elapsed_min": 179.0, "submitted": 3502, "finished": 3257, "in_flight": 245, "gen_tokens": 40556016, "tok_per_s": 3776, "chain_tokens_median": 1865.0, "chain_tokens_p90": 6370.900000000003, "chain_tokens_max": 32674, "parse_rate": 0.9888701258827142, "looped": 145, "minutes_to_deadline": -0.6} | |
| (MainProcess pid=64676) INFO 10-31 13:10:31 [loggers.py:335] Engine 000: Avg prompt throughput: 331.5 tokens/s, Avg generation throughput: 3787.3 tokens/s, Running: 256 reqs, Waiting: 593 reqs, GPU KV cache usage: 85.1%, Prefix cache hit rate: 44.0% | |
| progress {"time": "13:11:31", "elapsed_min": 180.0, "submitted": 3502, "finished": 3267, "in_flight": 235, "gen_tokens": 40689304, "tok_per_s": 3767, "chain_tokens_median": 1869.5, "chain_tokens_p90": 6370.900000000003, "chain_tokens_max": 32674, "parse_rate": 0.988904193449648, "looped": 145, "minutes_to_deadline": -1.6} | |
| (MainProcess pid=64676) INFO 10-31 13:11:31 [loggers.py:335] Engine 000: Avg prompt throughput: 311.9 tokens/s, Avg generation throughput: 3783.1 tokens/s, Running: 256 reqs, Waiting: 538 reqs, GPU KV cache usage: 88.5%, Prefix cache hit rate: 44.0% | |
| progress {"time": "13:12:31", "elapsed_min": 181.0, "submitted": 3502, "finished": 3286, "in_flight": 216, "gen_tokens": 40940198, "tok_per_s": 3770, "chain_tokens_median": 1870.0, "chain_tokens_p90": 6377.700000000001, "chain_tokens_max": 32674, "parse_rate": 0.9888922702373707, "looped": 146, "minutes_to_deadline": -2.6} | |
| (MainProcess pid=64676) INFO 10-31 13:12:31 [loggers.py:335] Engine 000: Avg prompt throughput: 275.9 tokens/s, Avg generation throughput: 3680.4 tokens/s, Running: 256 reqs, Waiting: 467 reqs, GPU KV cache usage: 88.0%, Prefix cache hit rate: 41.0% | |
| progress {"time": "13:13:31", "elapsed_min": 182.0, "submitted": 3502, "finished": 3300, "in_flight": 202, "gen_tokens": 41134521, "tok_per_s": 3767, "chain_tokens_median": 1865.0, "chain_tokens_p90": 6378.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9887878787878788, "looped": 148, "minutes_to_deadline": -3.6} | |
| (MainProcess pid=64676) INFO 10-31 13:13:31 [loggers.py:335] Engine 000: Avg prompt throughput: 377.2 tokens/s, Avg generation throughput: 3684.6 tokens/s, Running: 256 reqs, Waiting: 407 reqs, GPU KV cache usage: 89.0%, Prefix cache hit rate: 40.8% | |
| progress {"time": "13:14:31", "elapsed_min": 183.0, "submitted": 3502, "finished": 3324, "in_flight": 178, "gen_tokens": 41414406, "tok_per_s": 3771, "chain_tokens_median": 1870.5, "chain_tokens_p90": 6374.5, "chain_tokens_max": 32674, "parse_rate": 0.9888688327316486, "looped": 148, "minutes_to_deadline": -4.6} | |
| (MainProcess pid=64676) INFO 10-31 13:14:31 [loggers.py:335] Engine 000: Avg prompt throughput: 348.4 tokens/s, Avg generation throughput: 3633.4 tokens/s, Running: 256 reqs, Waiting: 333 reqs, GPU KV cache usage: 87.1%, Prefix cache hit rate: 40.4% | |
| progress {"time": "13:15:31", "elapsed_min": 184.0, "submitted": 3502, "finished": 3337, "in_flight": 165, "gen_tokens": 41642579, "tok_per_s": 3772, "chain_tokens_median": 1865.5, "chain_tokens_p90": 6373.900000000003, "chain_tokens_max": 32674, "parse_rate": 0.9886125262211567, "looped": 152, "minutes_to_deadline": -5.6} | |
| (MainProcess pid=64676) INFO 10-31 13:15:31 [loggers.py:335] Engine 000: Avg prompt throughput: 300.9 tokens/s, Avg generation throughput: 3753.1 tokens/s, Running: 256 reqs, Waiting: 275 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:16:31", "elapsed_min": 185.0, "submitted": 3502, "finished": 3355, "in_flight": 147, "gen_tokens": 41927645, "tok_per_s": 3777, "chain_tokens_median": 1866.5, "chain_tokens_p90": 6382.1, "chain_tokens_max": 32674, "parse_rate": 0.9885991058122205, "looped": 153, "minutes_to_deadline": -6.6} | |
| (MainProcess pid=64676) INFO 10-31 13:16:31 [loggers.py:335] Engine 000: Avg prompt throughput: 316.9 tokens/s, Avg generation throughput: 3684.5 tokens/s, Running: 255 reqs, Waiting: 198 reqs, GPU KV cache usage: 84.0%, Prefix cache hit rate: 39.6% | |
| progress {"time": "13:17:31", "elapsed_min": 186.0, "submitted": 3502, "finished": 3375, "in_flight": 127, "gen_tokens": 42335366, "tok_per_s": 3793, "chain_tokens_median": 1872.5, "chain_tokens_p90": 6404.1, "chain_tokens_max": 32674, "parse_rate": 0.9884444444444445, "looped": 156, "minutes_to_deadline": -7.6} | |
| (MainProcess pid=64676) INFO 10-31 13:17:31 [loggers.py:335] Engine 000: Avg prompt throughput: 292.5 tokens/s, Avg generation throughput: 3842.7 tokens/s, Running: 256 reqs, Waiting: 135 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 39.1% | |
| progress {"time": "13:18:31", "elapsed_min": 187.0, "submitted": 3502, "finished": 3389, "in_flight": 113, "gen_tokens": 42484672, "tok_per_s": 3786, "chain_tokens_median": 1872.5, "chain_tokens_p90": 6401.5, "chain_tokens_max": 32674, "parse_rate": 0.9884921805842432, "looped": 156, "minutes_to_deadline": -8.6} | |
| (MainProcess pid=64676) INFO 10-31 13:18:31 [loggers.py:335] Engine 000: Avg prompt throughput: 388.6 tokens/s, Avg generation throughput: 3842.5 tokens/s, Running: 256 reqs, Waiting: 64 reqs, GPU KV cache usage: 81.8%, Prefix cache hit rate: 39.4% | |
| progress {"time": "13:19:31", "elapsed_min": 188.0, "submitted": 3502, "finished": 3409, "in_flight": 93, "gen_tokens": 42640295, "tok_per_s": 3780, "chain_tokens_median": 1864.0, "chain_tokens_p90": 6386.5, "chain_tokens_max": 32674, "parse_rate": 0.988559694925198, "looped": 156, "minutes_to_deadline": -9.6} | |
| (MainProcess pid=64676) INFO 10-31 13:19:31 [loggers.py:335] Engine 000: Avg prompt throughput: 272.4 tokens/s, Avg generation throughput: 3855.1 tokens/s, Running: 251 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.4%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:20:31", "elapsed_min": 189.0, "submitted": 3502, "finished": 3426, "in_flight": 76, "gen_tokens": 42929995, "tok_per_s": 3785, "chain_tokens_median": 1870.0, "chain_tokens_p90": 6399.200000000004, "chain_tokens_max": 32674, "parse_rate": 0.988543490951547, "looped": 157, "minutes_to_deadline": -10.6} | |
| (MainProcess pid=64676) INFO 10-31 13:20:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3774.5 tokens/s, Running: 188 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.3%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:21:31", "elapsed_min": 190.0, "submitted": 3502, "finished": 3440, "in_flight": 62, "gen_tokens": 43237612, "tok_per_s": 3792, "chain_tokens_median": 1879.0, "chain_tokens_p90": 6410.0, "chain_tokens_max": 32674, "parse_rate": 0.9883720930232558, "looped": 160, "minutes_to_deadline": -11.6} | |
| (MainProcess pid=64676) INFO 10-31 13:21:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3144.2 tokens/s, Running: 126 reqs, Waiting: 0 reqs, GPU KV cache usage: 60.9%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:22:31", "elapsed_min": 191.0, "submitted": 3502, "finished": 3457, "in_flight": 45, "gen_tokens": 43696299, "tok_per_s": 3813, "chain_tokens_median": 1897.5, "chain_tokens_p90": 6458.5000000000055, "chain_tokens_max": 32674, "parse_rate": 0.9883569568990455, "looped": 161, "minutes_to_deadline": -12.6} | |
| (MainProcess pid=64676) INFO 10-31 13:22:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2589.3 tokens/s, Running: 76 reqs, Waiting: 0 reqs, GPU KV cache usage: 45.2%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:23:31", "elapsed_min": 192.0, "submitted": 3502, "finished": 3468, "in_flight": 34, "gen_tokens": 43945158, "tok_per_s": 3814, "chain_tokens_median": 1908.0, "chain_tokens_p90": 6469.9, "chain_tokens_max": 32674, "parse_rate": 0.9883217993079585, "looped": 162, "minutes_to_deadline": -13.6} | |
| (MainProcess pid=64676) INFO 10-31 13:23:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1917.1 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.1%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:24:31", "elapsed_min": 193.0, "submitted": 3502, "finished": 3480, "in_flight": 22, "gen_tokens": 44358298, "tok_per_s": 3830, "chain_tokens_median": 1922.5, "chain_tokens_p90": 6495.4000000000015, "chain_tokens_max": 32674, "parse_rate": 0.9880028735632184, "looped": 167, "minutes_to_deadline": -14.6} | |
| (MainProcess pid=64676) INFO 10-31 13:24:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1579.0 tokens/s, Running: 28 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:25:31", "elapsed_min": 194.0, "submitted": 3502, "finished": 3482, "in_flight": 20, "gen_tokens": 44420723, "tok_per_s": 3816, "chain_tokens_median": 1926.5, "chain_tokens_p90": 6499.0, "chain_tokens_max": 32674, "parse_rate": 0.9879379666858128, "looped": 168, "minutes_to_deadline": -15.6} | |
| (MainProcess pid=64676) INFO 10-31 13:25:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1376.2 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:26:31", "elapsed_min": 195.0, "submitted": 3502, "finished": 3484, "in_flight": 18, "gen_tokens": 44511945, "tok_per_s": 3804, "chain_tokens_median": 1931.5, "chain_tokens_p90": 6503.5, "chain_tokens_max": 32674, "parse_rate": 0.9878013777267508, "looped": 170, "minutes_to_deadline": -16.6} | |
| (MainProcess pid=64676) INFO 10-31 13:26:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1245.8 tokens/s, Running: 24 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.0%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:27:31", "elapsed_min": 196.0, "submitted": 3502, "finished": 3487, "in_flight": 15, "gen_tokens": 44703392, "tok_per_s": 3801, "chain_tokens_median": 1934.0, "chain_tokens_p90": 6507.300000000001, "chain_tokens_max": 32674, "parse_rate": 0.9874533983366791, "looped": 175, "minutes_to_deadline": -17.6} | |
| (MainProcess pid=64676) INFO 10-31 13:27:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1098.0 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.0%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:28:31", "elapsed_min": 197.0, "submitted": 3502, "finished": 3488, "in_flight": 14, "gen_tokens": 44749947, "tok_per_s": 3786, "chain_tokens_median": 1934.5, "chain_tokens_p90": 6507.9, "chain_tokens_max": 32674, "parse_rate": 0.9873853211009175, "looped": 176, "minutes_to_deadline": -18.6} | |
| (MainProcess pid=64676) INFO 10-31 13:28:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 976.1 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.1%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:29:31", "elapsed_min": 198.0, "submitted": 3502, "finished": 3492, "in_flight": 10, "gen_tokens": 44984627, "tok_per_s": 3786, "chain_tokens_median": 1940.0, "chain_tokens_p90": 6523.900000000003, "chain_tokens_max": 32674, "parse_rate": 0.9870418098510882, "looped": 181, "minutes_to_deadline": -19.6} | |
| (MainProcess pid=64676) INFO 10-31 13:29:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 845.3 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 39.3% | |
| progress {"time": "13:30:31", "elapsed_min": 199.0, "submitted": 3502, "finished": 3494, "in_flight": 8, "gen_tokens": 45086864, "tok_per_s": 3776, "chain_tokens_median": 1944.5, "chain_tokens_p90": 6529.5, "chain_tokens_max": 32674, "parse_rate": 0.9869061247853463, "looped": 183, "minutes_to_deadline": -20.6} | |
| (MainProcess pid=64676) INFO 10-31 13:30:31 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 728.5 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 39.3% | |
| (MainProcess pid=64676) INFO 10-31 13:31:15 [core_client.py:770] [shutdown] MPClient: start timeout=default | |
| (MainProcess pid=64676) INFO 10-31 13:31:15 [core_client.py:772] [shutdown] MPClient: stopping engine manager | |
| (MainProcess pid=64676) INFO 10-31 13:31:15 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=65100) INFO 10-31 13:31:15 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=65100) INFO 10-31 13:31:15 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=65100) INFO 10-31 13:31:15 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=65100) INFO 10-31 13:31:15 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=64676) WARNING 10-31 13:31:20 [utils.py:648] [shutdown] Process manager: force killing remaining processes count=1 | |
| (MainProcess pid=64676) WARNING 10-31 13:31:20 [utils.py:653] [shutdown] Process manager: force killing remaining process EngineCore pid 65100 | |
| (MainProcess pid=64676) INFO 10-31 13:31:20 [core_client.py:774] [shutdown] MPClient: engine manager stopped | |
| (MainProcess pid=64676) INFO 10-31 13:31:20 [core_client.py:775] [shutdown] MPClient: cleaning up background resources | |
| (MainProcess pid=64676) INFO 10-31 13:31:20 [core_client.py:777] [shutdown] MPClient: complete | |
| { | |
| "items": 6671, | |
| "samples": 26684, | |
| "tokens_total": 86522997, | |
| "tokens_median": 1948.0, | |
| "tokens_p90": 6593.700000000001, | |
| "tokens_max": 32674, | |
| "parse_rate": 0.9869584769899565, | |
| "looped": 348, | |
| "items_with_correct_chain": 5837, | |
| "majority_accuracy": 0.8316594213761055, | |
| "this_run": { | |
| "submitted": 3502, | |
| "finished": 3502, | |
| "gen_tokens": 45548964, | |
| "t0": 1793441490.3566904, | |
| "seconds": 11985.539092302322, | |
| "tok_per_s": 3800.3266811130497 | |
| }, | |
| "seconds_this_run": 12071.789855718613 | |
| } | |
| EXIT 0 | |
Xet Storage Details
- Size:
- 141 kB
- Xet hash:
- bc61c957a84ee2b2da45a4fe95b1af04558ba9a0d6a7d9ff548684af69f30b93
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.