Buckets:
| (MainProcess pid=207621) INFO 11-01 05:00:56 [model.py:733] Resolved architecture: LlamaForCausalLM | |
| (MainProcess pid=207621) INFO 11-01 05:00:56 [model.py:2152] Using max model len 32768 | |
| (MainProcess pid=207621) INFO 11-01 05:00:56 [scheduler.py:307] Chunked prefill is enabled with max_num_batched_tokens=2048. | |
| (MainProcess pid=207621) INFO 11-01 05:00:56 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (EngineCore pid=208645) INFO 11-01 05:00:59 [core.py:133] Initializing a V1 LLM engine (v0.32.0) with config: model='checkpoints/v2b_ro_merged', speculative_config=None, tokenizer='checkpoints/v2b_ro_merged', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=17, served_model_name=checkpoints/v2b_ro_merged, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::qwen4_exp_ple_short_conv', 'vllm::qwen4_exp_qsa_with_output', 'vllm::qwen4_exp_qsa_prepare_with_output', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 192, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None) | |
| (EngineCore pid=208645) INFO 11-01 05:01:00 [parallel_state.py:1906] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_466b7415ea9a47138686e330ca845e94 backend=nccl | |
| (EngineCore pid=208645) INFO 11-01 05:01:00 [parallel_state.py:2350] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A | |
| (EngineCore pid=208645) INFO 11-01 05:01:00 [gpu_worker.py:518] Using V2 Model Runner | |
| (EngineCore pid=208645) INFO 11-01 05:01:02 [model_runner.py:392] Loading model from scratch... | |
| (EngineCore pid=208645) INFO 11-01 05:01:03 [cuda.py:506] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. | |
| (EngineCore pid=208645) INFO 11-01 05:01:03 [flash_attn.py:1117] Using FlashAttention version 2 | |
| (EngineCore pid=208645) INFO 11-01 05:01:03 [base_loader.py:82] Initializing model took 1.451003 seconds | |
| (EngineCore pid=208645) INFO 11-01 05:01:03 [weight_utils.py:1108] Filesystem type for checkpoints: XFS. Checkpoint size: 4.69 GiB. Available RAM: 122.94 GiB. | |
| (EngineCore pid=208645) INFO 11-01 05:01:03 [weight_utils.py:1131] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. | |
| Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.96it/s] | |
| Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 1.96it/s] | |
| (EngineCore pid=208645) INFO 11-01 05:01:04 [default_loader.py:501] Loading weights took 0.53 seconds | |
| (EngineCore pid=208645) INFO 11-01 05:01:05 [model_runner.py:424] Model loading took 4.75 GiB memory and 3.708639 seconds | |
| (EngineCore pid=208645) INFO 11-01 05:01:05 [topk_topp_sampler.py:124] Using FlashInfer for top-p & top-k sampling. | |
| (EngineCore pid=208645) INFO 11-01 05:01:05 [torch_utils.py:295] Reducing Torch threads from 128 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override. | |
| (EngineCore pid=208645) INFO 11-01 05:01:05 [utils.py:347] Using LBNHC KV cache layout. | |
| (EngineCore pid=208645) INFO 11-01 05:01:06 [monitor.py:40] [backbone] Starting torch.compile | |
| (EngineCore pid=208645) INFO 11-01 05:01:15 [backends.py:1104] [backbone] Using cache directory: /root/.cache/vllm/torch_compile_cache/fcd2c507ca/rank_0_0/backbone for vLLM's torch.compile | |
| (EngineCore pid=208645) INFO 11-01 05:01:15 [backends.py:1166] [backbone] Dynamo bytecode transform time: 9.13 s | |
| (EngineCore pid=208645) INFO 11-01 05:01:24 [backends.py:398] [backbone] Compiling a graph for compile range (1, 2048) takes 8.90 s | |
| (EngineCore pid=208645) INFO 11-01 05:01:30 [backends.py:927] collected artifacts: 43 entries, 3 artifacts, 4889756 bytes total | |
| (EngineCore pid=208645) INFO 11-01 05:01:30 [decorators.py:743] [backbone] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/f9c866bc30b2755a5fb65bc02c0d3dfb626af1bb088bfd085f865d9bd4b40eed/rank_0_0/model | |
| (EngineCore pid=208645) INFO 11-01 05:01:30 [monitor.py:62] [backbone] torch.compile took 23.87 s in total | |
| (EngineCore pid=208645) INFO 11-01 05:01:30 [monitor.py:91] [backbone] Initial profiling/warmup run took 0.20 s | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/27 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 7%|▋ | 2/27 [00:00<00:01, 17.69it/s] Capturing CUDA graphs (PIECEWISE): 19%|█▊ | 5/27 [00:00<00:01, 21.08it/s] Capturing CUDA graphs (PIECEWISE): 30%|██▉ | 8/27 [00:00<00:00, 22.11it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 11/27 [00:00<00:00, 21.79it/s] Capturing CUDA graphs (PIECEWISE): 52%|█████▏ | 14/27 [00:00<00:00, 20.95it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 17/27 [00:00<00:00, 20.84it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 20/27 [00:00<00:00, 20.56it/s] Capturing CUDA graphs (PIECEWISE): 85%|████████▌ | 23/27 [00:01<00:00, 20.65it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▋| 26/27 [00:01<00:00, 19.71it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:01<00:00, 20.21it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/2 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 100%|██████████| 2/2 [00:00<00:00, 26.36it/s] | |
| (EngineCore pid=208645) INFO 11-01 05:01:34 [model_runner.py:1127] Graph capturing finished in 2 secs, took 0.24 GiB | |
| (EngineCore pid=208645) INFO 11-01 05:01:35 [gpu_worker.py:759] Available KV cache memory: 67.09 GiB | |
| (EngineCore pid=208645) INFO 11-01 05:01:35 [gpu_worker.py:774] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9164 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9236. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. | |
| (EngineCore pid=208645) INFO 11-01 05:01:35 [kv_cache_utils.py:2327] GPU KV cache size: 1,675,072 tokens, Maximum concurrency for 32,768 tokens per request: 51.12x | |
| (EngineCore pid=208645) INFO 11-01 05:01:35 [kernel_warmup.py:176] JIT kernel warmup starting. | |
| (EngineCore pid=208645) INFO 11-01 05:01:35 [kernel_warmup.py:189] JIT kernel warmup finished in 0.00s. | |
| Capturing CUDA graphs (PIECEWISE): 0%| | 0/27 [00:00<?, ?it/s] Capturing CUDA graphs (PIECEWISE): 7%|▋ | 2/27 [00:00<00:01, 16.76it/s] Capturing CUDA graphs (PIECEWISE): 19%|█▊ | 5/27 [00:00<00:01, 20.46it/s] Capturing CUDA graphs (PIECEWISE): 30%|██▉ | 8/27 [00:00<00:00, 21.59it/s] Capturing CUDA graphs (PIECEWISE): 41%|████ | 11/27 [00:00<00:00, 22.16it/s] Capturing CUDA graphs (PIECEWISE): 52%|█████▏ | 14/27 [00:00<00:00, 21.30it/s] Capturing CUDA graphs (PIECEWISE): 63%|██████▎ | 17/27 [00:00<00:00, 21.16it/s] Capturing CUDA graphs (PIECEWISE): 74%|███████▍ | 20/27 [00:00<00:00, 21.01it/s] Capturing CUDA graphs (PIECEWISE): 85%|████████▌ | 23/27 [00:01<00:00, 20.85it/s] Capturing CUDA graphs (PIECEWISE): 96%|█████████▋| 26/27 [00:01<00:00, 20.58it/s] Capturing CUDA graphs (PIECEWISE): 100%|██████████| 27/27 [00:01<00:00, 20.87it/s] | |
| Capturing CUDA graphs (FULL): 0%| | 0/15 [00:00<?, ?it/s] Capturing CUDA graphs (FULL): 20%|██ | 3/15 [00:00<00:00, 23.82it/s] Capturing CUDA graphs (FULL): 40%|████ | 6/15 [00:00<00:00, 24.16it/s] Capturing CUDA graphs (FULL): 60%|██████ | 9/15 [00:00<00:00, 24.75it/s] Capturing CUDA graphs (FULL): 80%|████████ | 12/15 [00:00<00:00, 25.20it/s] Capturing CUDA graphs (FULL): 100%|██████████| 15/15 [00:00<00:00, 25.51it/s] Capturing CUDA graphs (FULL): 100%|██████████| 15/15 [00:00<00:00, 25.10it/s] | |
| (EngineCore pid=208645) INFO 11-01 05:01:55 [model_runner.py:1127] Graph capturing finished in 3 secs, took 0.12 GiB | |
| (EngineCore pid=208645) INFO 11-01 05:01:55 [gpu_worker.py:971] CUDA graph pool memory: 0.12 GiB (actual), 0.29 GiB (estimated), difference: 0.17 GiB (146.7%). | |
| (EngineCore pid=208645) INFO 11-01 05:01:55 [gpu_worker.py:1034] Free memory on device (78.84/79.25 GiB) on startup. Desired GPU memory utilization is (0.92, 72.91 GiB). Actual usage is 5.02 GiB for consumed memory (weights + non-torch), 0.79 GiB for peak activation, and 0.12 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=71758921954` (66.83 GiB) to fit into requested memory, or `--kv-cache-memory=78123091456` (72.76 GiB) to fully utilize gpu memory. Current kv cache memory in use is 67.09 GiB. | |
| (EngineCore pid=208645) INFO 11-01 05:01:55 [jit_monitor.py:84] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. | |
| (EngineCore pid=208645) INFO 11-01 05:01:56 [core.py:395] init engine (profile, create kv cache, warmup model) took 50.78 s (compilation: 23.87 s) | |
| (EngineCore pid=208645) INFO 11-01 05:01:56 [kernel.py:467] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'], gelu_and_mul_sparse=['triton', 'native']) | |
| (MainProcess pid=207621) WARNING 11-01 05:01:57 [input_processor.py:408] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat(). | |
| (EngineCore pid=208645) WARNING 11-01 05:01:57 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topk_log_softmax_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| readout v2b_ro: 5957 items in 48s | |
| (EngineCore pid=208645) WARNING 11-01 05:02:45 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=208645) WARNING 11-01 05:02:46 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_step_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| (EngineCore pid=208645) WARNING 11-01 05:02:47 [jit_monitor.py:140] Triton kernel JIT compilation during inference: _topp_sb_mask_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. | |
| progress {"time": "05:03:44", "elapsed_min": 1.0, "submitted": 911, "finished": 655, "in_flight": 256, "gen_tokens": 138228, "tok_per_s": 2304, "chain_tokens_median": 85.0, "chain_tokens_p90": 586.6000000000001, "chain_tokens_max": 2293, "parse_rate": 0.981679389312977, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:03:44 [loggers.py:335] Engine 000: Avg prompt throughput: 21538.6 tokens/s, Avg generation throughput: 2220.5 tokens/s, Running: 96 reqs, Waiting: 160 reqs, GPU KV cache usage: 8.4%, Prefix cache hit rate: 69.4% | |
| progress {"time": "05:04:44", "elapsed_min": 2.0, "submitted": 1240, "finished": 984, "in_flight": 256, "gen_tokens": 254052, "tok_per_s": 2117, "chain_tokens_median": 90.5, "chain_tokens_p90": 707.7, "chain_tokens_max": 4591, "parse_rate": 0.9867886178861789, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:04:44 [loggers.py:335] Engine 000: Avg prompt throughput: 92.6 tokens/s, Avg generation throughput: 4211.1 tokens/s, Running: 96 reqs, Waiting: 159 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 90.1% | |
| progress {"time": "05:05:44", "elapsed_min": 3.0, "submitted": 1565, "finished": 1309, "in_flight": 256, "gen_tokens": 373755, "tok_per_s": 2076, "chain_tokens_median": 101.0, "chain_tokens_p90": 722.4000000000001, "chain_tokens_max": 4846, "parse_rate": 0.985485103132162, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:05:44 [loggers.py:335] Engine 000: Avg prompt throughput: 96.2 tokens/s, Avg generation throughput: 3048.1 tokens/s, Running: 96 reqs, Waiting: 160 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 98.4% | |
| progress {"time": "05:06:44", "elapsed_min": 4.0, "submitted": 1747, "finished": 1491, "in_flight": 256, "gen_tokens": 446165, "tok_per_s": 1859, "chain_tokens_median": 112.0, "chain_tokens_p90": 724.0, "chain_tokens_max": 6443, "parse_rate": 0.9852448021462106, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:06:44 [loggers.py:335] Engine 000: Avg prompt throughput: 58.3 tokens/s, Avg generation throughput: 2862.9 tokens/s, Running: 96 reqs, Waiting: 160 reqs, GPU KV cache usage: 25.6%, Prefix cache hit rate: 98.7% | |
| progress {"time": "05:07:44", "elapsed_min": 5.0, "submitted": 1908, "finished": 1652, "in_flight": 256, "gen_tokens": 492969, "tok_per_s": 1643, "chain_tokens_median": 110.0, "chain_tokens_p90": 724.9000000000001, "chain_tokens_max": 7997, "parse_rate": 0.9830508474576272, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:07:44 [loggers.py:335] Engine 000: Avg prompt throughput: 851.7 tokens/s, Avg generation throughput: 2367.2 tokens/s, Running: 96 reqs, Waiting: 159 reqs, GPU KV cache usage: 30.7%, Prefix cache hit rate: 91.8% | |
| progress {"time": "05:08:44", "elapsed_min": 6.0, "submitted": 2013, "finished": 1804, "in_flight": 209, "gen_tokens": 548977, "tok_per_s": 1525, "chain_tokens_median": 110.5, "chain_tokens_p90": 749.7, "chain_tokens_max": 7997, "parse_rate": 0.9822616407982262, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:08:44 [loggers.py:335] Engine 000: Avg prompt throughput: 647.3 tokens/s, Avg generation throughput: 2270.7 tokens/s, Running: 96 reqs, Waiting: 113 reqs, GPU KV cache usage: 36.3%, Prefix cache hit rate: 86.7% | |
| progress {"time": "05:09:44", "elapsed_min": 7.0, "submitted": 2013, "finished": 1932, "in_flight": 81, "gen_tokens": 617537, "tok_per_s": 1470, "chain_tokens_median": 114.0, "chain_tokens_p90": 778.9000000000001, "chain_tokens_max": 7997, "parse_rate": 0.9813664596273292, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:09:44 [loggers.py:335] Engine 000: Avg prompt throughput: 325.0 tokens/s, Avg generation throughput: 2270.6 tokens/s, Running: 81 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.0%, Prefix cache hit rate: 84.5% | |
| progress {"time": "05:10:44", "elapsed_min": 8.0, "submitted": 2013, "finished": 1944, "in_flight": 69, "gen_tokens": 651925, "tok_per_s": 1358, "chain_tokens_median": 115.5, "chain_tokens_p90": 813.7, "chain_tokens_max": 7997, "parse_rate": 0.9814814814814815, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:10:44 [loggers.py:335] Engine 000: Avg prompt throughput: 1.7 tokens/s, Avg generation throughput: 2693.9 tokens/s, Running: 69 reqs, Waiting: 0 reqs, GPU KV cache usage: 48.0%, Prefix cache hit rate: 85.2% | |
| progress {"time": "05:11:44", "elapsed_min": 9.0, "submitted": 2013, "finished": 1945, "in_flight": 68, "gen_tokens": 656756, "tok_per_s": 1216, "chain_tokens_median": 116.0, "chain_tokens_p90": 816.4000000000005, "chain_tokens_max": 7997, "parse_rate": 0.9814910025706941, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:11:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.1 tokens/s, Avg generation throughput: 2263.6 tokens/s, Running: 68 reqs, Waiting: 0 reqs, GPU KV cache usage: 55.8%, Prefix cache hit rate: 85.3% | |
| progress {"time": "05:12:44", "elapsed_min": 10.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 1107, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:12:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.1 tokens/s, Avg generation throughput: 2025.9 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 62.6%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:13:44", "elapsed_min": 11.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 1006, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:13:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1859.1 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.2%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:14:44", "elapsed_min": 12.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 922, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:14:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1728.5 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.4%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:15:44", "elapsed_min": 13.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 851, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:15:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1620.2 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.2%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:16:44", "elapsed_min": 14.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 790, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:16:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1532.0 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.7%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:17:44", "elapsed_min": 15.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 738, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:17:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1455.0 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.9%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:18:44", "elapsed_min": 16.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 692, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:18:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1390.2 tokens/s, Running: 67 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.9%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:19:44", "elapsed_min": 17.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 651, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:19:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1328.8 tokens/s, Running: 65 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:20:44", "elapsed_min": 18.0, "submitted": 2013, "finished": 1946, "in_flight": 67, "gen_tokens": 664007, "tok_per_s": 615, "chain_tokens_median": 116.0, "chain_tokens_p90": 818.5, "chain_tokens_max": 7997, "parse_rate": 0.9815005138746146, "looped": 0, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:20:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1252.1 tokens/s, Running: 61 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:21:44", "elapsed_min": 19.0, "submitted": 2013, "finished": 1947, "in_flight": 66, "gen_tokens": 694152, "tok_per_s": 609, "chain_tokens_median": 116.0, "chain_tokens_p90": 819.0, "chain_tokens_max": 30145, "parse_rate": 0.9809964047252183, "looped": 1, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:21:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1005.1 tokens/s, Running: 59 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.4%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:22:44", "elapsed_min": 20.0, "submitted": 2013, "finished": 1949, "in_flight": 64, "gen_tokens": 757380, "tok_per_s": 631, "chain_tokens_median": 116.0, "chain_tokens_p90": 822.4000000000001, "chain_tokens_max": 31639, "parse_rate": 0.9799897383273474, "looped": 3, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:22:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 999.3 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.0%, Prefix cache hit rate: 85.4% | |
| progress {"time": "05:23:44", "elapsed_min": 21.0, "submitted": 2013, "finished": 1957, "in_flight": 56, "gen_tokens": 1013726, "tok_per_s": 805, "chain_tokens_median": 117.0, "chain_tokens_p90": 844.6000000000004, "chain_tokens_max": 32562, "parse_rate": 0.9785385794583547, "looped": 6, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:23:44 [loggers.py:335] Engine 000: Avg prompt throughput: 54.8 tokens/s, Avg generation throughput: 771.2 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 94.6%, Prefix cache hit rate: 85.0% | |
| progress {"time": "05:24:44", "elapsed_min": 22.0, "submitted": 2013, "finished": 1981, "in_flight": 32, "gen_tokens": 1789294, "tok_per_s": 1355, "chain_tokens_median": 121.0, "chain_tokens_p90": 917.0, "chain_tokens_max": 32633, "parse_rate": 0.9702170620898536, "looped": 23, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:24:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.7 tokens/s, Avg generation throughput: 1109.1 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 51.7%, Prefix cache hit rate: 84.9% | |
| progress {"time": "05:25:44", "elapsed_min": 23.0, "submitted": 2013, "finished": 1991, "in_flight": 22, "gen_tokens": 2109667, "tok_per_s": 1529, "chain_tokens_median": 121.0, "chain_tokens_p90": 960.0, "chain_tokens_max": 32634, "parse_rate": 0.9678553490708187, "looped": 28, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:25:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.9 tokens/s, Avg generation throughput: 1015.3 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.8%, Prefix cache hit rate: 84.8% | |
| progress {"time": "05:26:44", "elapsed_min": 24.0, "submitted": 2013, "finished": 1995, "in_flight": 18, "gen_tokens": 2238611, "tok_per_s": 1555, "chain_tokens_median": 121.0, "chain_tokens_p90": 988.6000000000015, "chain_tokens_max": 32634, "parse_rate": 0.9659147869674185, "looped": 32, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:26:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 934.0 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.3%, Prefix cache hit rate: 84.8% | |
| progress {"time": "05:27:44", "elapsed_min": 25.0, "submitted": 2013, "finished": 2006, "in_flight": 7, "gen_tokens": 2593128, "tok_per_s": 1729, "chain_tokens_median": 123.0, "chain_tokens_p90": 1035.5, "chain_tokens_max": 32634, "parse_rate": 0.9621136590229312, "looped": 40, "minutes_to_deadline": Infinity} | |
| (MainProcess pid=207621) INFO 11-01 05:27:44 [loggers.py:335] Engine 000: Avg prompt throughput: 0.5 tokens/s, Avg generation throughput: 737.5 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 84.8% | |
| reason v2b_ro: {"submitted": 2013, "finished": 2013, "gen_tokens": 2818892, "t0": 1793509364.7541668, "seconds": 1550.2733824253082, "tok_per_s": 1818.3192925560106} | |
| reason v2b_ro: post-chain readout 1970 taken at generation, 0 by prefill | |
| reason v2b_ro: 2013 items, post-readout in 1550s total | |
| (MainProcess pid=207621) INFO 11-01 05:28:35 [core_client.py:770] [shutdown] MPClient: start timeout=default | |
| (MainProcess pid=207621) INFO 11-01 05:28:35 [core_client.py:772] [shutdown] MPClient: stopping engine manager | |
| (MainProcess pid=207621) INFO 11-01 05:28:35 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore | |
| (EngineCore pid=208645) INFO 11-01 05:28:35 [core.py:1454] [shutdown] EngineCore: trigger received signal=SIGTERM | |
| (EngineCore pid=208645) INFO 11-01 05:28:35 [core.py:1635] [shutdown] EngineCore: start mode=abort timeout=0s | |
| (EngineCore pid=208645) INFO 11-01 05:28:35 [core.py:1666] [shutdown] EngineCore: request processing complete; starting resource teardown | |
| (EngineCore pid=208645) INFO 11-01 05:28:35 [core.py:1499] [shutdown] EngineCore: exiting busy loop | |
| (MainProcess pid=207621) INFO 11-01 05:28:37 [core_client.py:774] [shutdown] MPClient: engine manager stopped | |
| (MainProcess pid=207621) INFO 11-01 05:28:37 [core_client.py:775] [shutdown] MPClient: cleaning up background resources | |
| (MainProcess pid=207621) INFO 11-01 05:28:37 [core_client.py:777] [shutdown] MPClient: complete | |
Xet Storage Details
- Size:
- 29.9 kB
- Xet hash:
- 5228ad1cd4768ad8ccdd341f05862a253a583f88d11a9cd633f2ae8dfbd2163d
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.