journal flashnext
Browse files- etat/3jzaxcgvs3kf7t.log +43 -40
etat/3jzaxcgvs3kf7t.log
CHANGED
|
@@ -1,51 +1,14 @@
|
|
| 1 |
-
=== lancer_flashnext | 19:
|
| 2 |
97056 MiB, 97887 MiB
|
| 3 |
total used free shared buff/cache available
|
| 4 |
-
Mem: 1511
|
| 5 |
--- comptabilite VRAM ---
|
| 6 |
(Worker pid=1040) INFO 09-05 17:56:26 [model_runner.py:407] Model loading took 74.47 GiB memory and 134.487932 seconds
|
| 7 |
(Worker pid=1040) INFO 09-05 17:57:38 [gpu_worker.py:621] Initial free memory 94.43 GiB, reserved 14.5 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly.
|
| 8 |
(EngineCore pid=819) INFO 09-05 17:57:38 [kv_cache_utils.py:2258] GPU KV cache size: 604,705 tokens, Maximum concurrency for 524,288 tokens per request: 1.15x
|
| 9 |
(Worker pid=1040) INFO 09-05 17:57:59 [model_runner.py:953] Graph capturing finished in 5 secs, took 0.21 GiB
|
| 10 |
--- vllm.log (fin) ---
|
| 11 |
-
reqs, GPU KV cache usage:
|
| 12 |
-
(APIServer pid=83) INFO 09-05 18:44:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 48.2%, Prefix cache hit rate: 32.1%
|
| 13 |
-
(APIServer pid=83) INFO: 127.0.0.1:46052 - "GET /v1/models HTTP/1.1" 200 OK
|
| 14 |
-
(APIServer pid=83) INFO 09-05 18:44:25 [loggers.py:310] Engine 000: Avg prompt throughput: 21033.4 tokens/s, Avg generation throughput: 18.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 63.8%, Prefix cache hit rate: 32.1%
|
| 15 |
-
(APIServer pid=83) INFO 09-05 18:44:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 85.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.0%, Prefix cache hit rate: 32.1%
|
| 16 |
-
(APIServer pid=83) INFO: 100.64.1.3:47208 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 17 |
-
(APIServer pid=83) INFO 09-05 18:44:45 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.3%, Prefix cache hit rate: 32.1%
|
| 18 |
-
(APIServer pid=83) INFO 09-05 18:44:55 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.7%, Prefix cache hit rate: 32.1%
|
| 19 |
-
(APIServer pid=83) INFO 09-05 18:45:05 [loggers.py:310] Engine 000: Avg prompt throughput: 21164.9 tokens/s, Avg generation throughput: 37.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.2%, Prefix cache hit rate: 32.1%
|
| 20 |
-
(APIServer pid=83) INFO 09-05 18:45:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 84.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.4%, Prefix cache hit rate: 32.1%
|
| 21 |
-
(APIServer pid=83) INFO 09-05 18:45:25 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 85.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.5%, Prefix cache hit rate: 32.1%
|
| 22 |
-
(APIServer pid=83) INFO 09-05 18:45:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 84.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.6%, Prefix cache hit rate: 32.1%
|
| 23 |
-
(APIServer pid=83) INFO 09-05 18:45:45 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 72.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 32.1%
|
| 24 |
-
(APIServer pid=83) INFO: 100.64.1.5:32882 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 25 |
-
(APIServer pid=83) INFO 09-05 18:45:55 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.7%, Prefix cache hit rate: 32.1%
|
| 26 |
-
(APIServer pid=83) INFO 09-05 18:46:15 [loggers.py:310] Engine 000: Avg prompt throughput: 21540.2 tokens/s, Avg generation throughput: 65.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 65.4%, Prefix cache hit rate: 32.1%
|
| 27 |
-
(APIServer pid=83) INFO: 100.64.1.3:49770 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 28 |
-
(APIServer pid=83) INFO 09-05 18:46:25 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.1%, Prefix cache hit rate: 32.1%
|
| 29 |
-
(APIServer pid=83) INFO 09-05 18:46:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 46.2%, Prefix cache hit rate: 32.1%
|
| 30 |
-
(APIServer pid=83) INFO 09-05 18:46:45 [loggers.py:310] Engine 000: Avg prompt throughput: 21660.8 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 65.4%, Prefix cache hit rate: 32.1%
|
| 31 |
-
(APIServer pid=83) INFO: 100.64.1.5:36726 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 32 |
-
(APIServer pid=83) INFO 09-05 18:46:55 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.3%, Prefix cache hit rate: 32.1%
|
| 33 |
-
(APIServer pid=83) INFO 09-05 18:47:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.7%, Prefix cache hit rate: 32.1%
|
| 34 |
-
(APIServer pid=83) INFO 09-05 18:47:15 [loggers.py:310] Engine 000: Avg prompt throughput: 21693.5 tokens/s, Avg generation throughput: 34.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 65.5%, Prefix cache hit rate: 32.1%
|
| 35 |
-
(APIServer pid=83) INFO: 100.64.1.2:60958 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 36 |
-
(APIServer pid=83) INFO 09-05 18:47:25 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 32.1%
|
| 37 |
-
(APIServer pid=83) INFO 09-05 18:47:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.0%, Prefix cache hit rate: 32.1%
|
| 38 |
-
(APIServer pid=83) INFO 09-05 18:47:45 [loggers.py:310] Engine 000: Avg prompt throughput: 21749.0 tokens/s, Avg generation throughput: 16.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 65.7%, Prefix cache hit rate: 32.1%
|
| 39 |
-
(APIServer pid=83) INFO: 100.64.1.4:39936 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 40 |
-
(APIServer pid=83) INFO 09-05 18:47:55 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.3%, Prefix cache hit rate: 32.0%
|
| 41 |
-
(APIServer pid=83) INFO 09-05 18:48:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 55.6%, Prefix cache hit rate: 32.0%
|
| 42 |
-
(APIServer pid=83) INFO 09-05 18:48:15 [loggers.py:310] Engine 000: Avg prompt throughput: 21786.8 tokens/s, Avg generation throughput: 42.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 65.7%, Prefix cache hit rate: 32.0%
|
| 43 |
-
(APIServer pid=83) INFO: 100.64.1.4:39936 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 44 |
-
(APIServer pid=83) INFO 09-05 18:48:25 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.5%, Prefix cache hit rate: 32.0%
|
| 45 |
-
(APIServer pid=83) INFO 09-05 18:48:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 51.8%, Prefix cache hit rate: 32.0%
|
| 46 |
-
(APIServer pid=83) INFO 09-05 18:48:45 [loggers.py:310] Engine 000: Avg prompt throughput: 21888.3 tokens/s, Avg generation throughput: 25.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 65.8%, Prefix cache hit rate: 32.0%
|
| 47 |
-
(APIServer pid=83) INFO 09-05 18:48:55 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 85.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 65.9%, Prefix cache hit rate: 32.0%
|
| 48 |
-
(APIServer pid=83) INFO 09-05 18:49:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 84.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 66.2%, Prefix cache hit rate: 32.0%
|
| 49 |
(APIServer pid=83) INFO 09-05 18:49:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 77.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 32.0%
|
| 50 |
(APIServer pid=83) INFO: 100.64.1.2:41354 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 51 |
(APIServer pid=83) INFO: 127.0.0.1:51270 - "GET /v1/models HTTP/1.1" 200 OK
|
|
@@ -215,4 +178,44 @@ reqs, GPU KV cache usage: 25.8%, Prefix cache hit rate: 32.1%
|
|
| 215 |
(APIServer pid=83) INFO 09-05 19:14:25 [loggers.py:310] Engine 000: Avg prompt throughput: 25600.0 tokens/s, Avg generation throughput: 53.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.9%, Prefix cache hit rate: 30.5%
|
| 216 |
(APIServer pid=83) INFO: 100.64.1.3:40114 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 217 |
(APIServer pid=83) INFO 09-05 19:14:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.5%, Prefix cache hit rate: 30.5%
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 218 |
--- pont.log (fin) ---
|
|
|
|
| 1 |
+
=== lancer_flashnext | 19:19:46 UTC ===
|
| 2 |
97056 MiB, 97887 MiB
|
| 3 |
total used free shared buff/cache available
|
| 4 |
+
Mem: 1511 242 456 1 821 1268
|
| 5 |
--- comptabilite VRAM ---
|
| 6 |
(Worker pid=1040) INFO 09-05 17:56:26 [model_runner.py:407] Model loading took 74.47 GiB memory and 134.487932 seconds
|
| 7 |
(Worker pid=1040) INFO 09-05 17:57:38 [gpu_worker.py:621] Initial free memory 94.43 GiB, reserved 14.5 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly.
|
| 8 |
(EngineCore pid=819) INFO 09-05 17:57:38 [kv_cache_utils.py:2258] GPU KV cache size: 604,705 tokens, Maximum concurrency for 524,288 tokens per request: 1.15x
|
| 9 |
(Worker pid=1040) INFO 09-05 17:57:59 [model_runner.py:953] Graph capturing finished in 5 secs, took 0.21 GiB
|
| 10 |
--- vllm.log (fin) ---
|
| 11 |
+
9:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 84.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 66.2%, Prefix cache hit rate: 32.0%
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
(APIServer pid=83) INFO 09-05 18:49:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 77.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 32.0%
|
| 13 |
(APIServer pid=83) INFO: 100.64.1.2:41354 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 14 |
(APIServer pid=83) INFO: 127.0.0.1:51270 - "GET /v1/models HTTP/1.1" 200 OK
|
|
|
|
| 178 |
(APIServer pid=83) INFO 09-05 19:14:25 [loggers.py:310] Engine 000: Avg prompt throughput: 25600.0 tokens/s, Avg generation throughput: 53.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.9%, Prefix cache hit rate: 30.5%
|
| 179 |
(APIServer pid=83) INFO: 100.64.1.3:40114 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 180 |
(APIServer pid=83) INFO 09-05 19:14:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.5%, Prefix cache hit rate: 30.5%
|
| 181 |
+
(APIServer pid=83) INFO 09-05 19:14:45 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 57.8%, Prefix cache hit rate: 30.5%
|
| 182 |
+
(APIServer pid=83) INFO: 127.0.0.1:59502 - "GET /v1/models HTTP/1.1" 200 OK
|
| 183 |
+
(APIServer pid=83) INFO 09-05 19:14:55 [loggers.py:310] Engine 000: Avg prompt throughput: 25741.8 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.1%, Prefix cache hit rate: 30.5%
|
| 184 |
+
(APIServer pid=83) INFO: 100.64.1.5:47058 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 185 |
+
(APIServer pid=83) INFO 09-05 19:15:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 60.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 30.4%
|
| 186 |
+
(APIServer pid=83) INFO: 100.64.1.2:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
|
| 187 |
+
(APIServer pid=83) INFO 09-05 19:15:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 1 reqs, GPU KV cache usage: 44.8%, Prefix cache hit rate: 30.4%
|
| 188 |
+
(APIServer pid=83) INFO: 100.64.1.2:60110 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 189 |
+
(APIServer pid=83) INFO 09-05 19:15:35 [loggers.py:310] Engine 000: Avg prompt throughput: 25815.7 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.6%, Prefix cache hit rate: 30.2%
|
| 190 |
+
(APIServer pid=83) INFO 09-05 19:15:45 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 1 reqs, GPU KV cache usage: 35.5%, Prefix cache hit rate: 30.2%
|
| 191 |
+
(APIServer pid=83) INFO 09-05 19:15:55 [loggers.py:310] Engine 000: Avg prompt throughput: 24910.9 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 1 reqs, Waiting: 1 reqs, GPU KV cache usage: 56.5%, Prefix cache hit rate: 30.2%
|
| 192 |
+
(APIServer pid=83) INFO 09-05 19:16:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.2%, Prefix cache hit rate: 30.2%
|
| 193 |
+
(APIServer pid=83) INFO 09-05 19:16:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 61.5%, Prefix cache hit rate: 30.2%
|
| 194 |
+
(APIServer pid=83) INFO 09-05 19:16:25 [loggers.py:310] Engine 000: Avg prompt throughput: 25856.4 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.3%, Prefix cache hit rate: 30.2%
|
| 195 |
+
(APIServer pid=83) INFO: 100.64.1.3:54988 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 196 |
+
(APIServer pid=83) INFO 09-05 19:16:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.5%, Prefix cache hit rate: 30.2%
|
| 197 |
+
(APIServer pid=83) INFO: 100.64.1.3:51786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
|
| 198 |
+
(APIServer pid=83) INFO 09-05 19:16:45 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 57.8%, Prefix cache hit rate: 30.2%
|
| 199 |
+
(APIServer pid=83) INFO 09-05 19:16:55 [loggers.py:310] Engine 000: Avg prompt throughput: 25729.1 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.4%, Prefix cache hit rate: 30.1%
|
| 200 |
+
(APIServer pid=83) INFO 09-05 19:17:05 [loggers.py:310] Engine 000: Avg prompt throughput: 6234.7 tokens/s, Avg generation throughput: 53.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 30.1%
|
| 201 |
+
(APIServer pid=83) INFO: 100.64.1.4:47754 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 202 |
+
(APIServer pid=83) INFO 09-05 19:17:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.2%, Prefix cache hit rate: 30.1%
|
| 203 |
+
(APIServer pid=83) INFO: 100.64.1.3:39462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
|
| 204 |
+
(APIServer pid=83) INFO 09-05 19:17:35 [loggers.py:310] Engine 000: Avg prompt throughput: 26019.2 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.1%, Prefix cache hit rate: 30.0%
|
| 205 |
+
(APIServer pid=83) INFO 09-05 19:17:45 [loggers.py:310] Engine 000: Avg prompt throughput: 12460.7 tokens/s, Avg generation throughput: 15.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.2%, Prefix cache hit rate: 30.0%
|
| 206 |
+
(APIServer pid=83) INFO: 100.64.1.4:47754 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 207 |
+
(APIServer pid=83) INFO 09-05 19:17:55 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 31.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 30.0%
|
| 208 |
+
(APIServer pid=83) INFO 09-05 19:18:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.2%, Prefix cache hit rate: 30.0%
|
| 209 |
+
(APIServer pid=83) INFO 09-05 19:18:25 [loggers.py:310] Engine 000: Avg prompt throughput: 26063.2 tokens/s, Avg generation throughput: 63.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 30.0%
|
| 210 |
+
(APIServer pid=83) INFO: 100.64.1.5:47272 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 211 |
+
(APIServer pid=83) INFO 09-05 19:18:35 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.2%, Prefix cache hit rate: 30.0%
|
| 212 |
+
(APIServer pid=83) INFO 09-05 19:18:55 [loggers.py:310] Engine 000: Avg prompt throughput: 26132.1 tokens/s, Avg generation throughput: 14.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.7%, Prefix cache hit rate: 30.0%
|
| 213 |
+
(APIServer pid=83) INFO: 100.64.1.4:48924 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK
|
| 214 |
+
(APIServer pid=83) INFO: 100.64.1.4:48924 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 215 |
+
(APIServer pid=83) INFO 09-05 19:19:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 28.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.1%, Prefix cache hit rate: 29.9%
|
| 216 |
+
(APIServer pid=83) INFO: 100.64.1.1:53126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
|
| 217 |
+
(APIServer pid=83) INFO 09-05 19:19:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 1 reqs, GPU KV cache usage: 52.2%, Prefix cache hit rate: 29.9%
|
| 218 |
+
(APIServer pid=83) INFO: 100.64.1.2:39994 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
|
| 219 |
+
(APIServer pid=83) INFO 09-05 19:19:35 [loggers.py:310] Engine 000: Avg prompt throughput: 26507.6 tokens/s, Avg generation throughput: 36.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 29.7%
|
| 220 |
+
(APIServer pid=83) INFO 09-05 19:19:45 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 1 reqs, GPU KV cache usage: 37.3%, Prefix cache hit rate: 29.7%
|
| 221 |
--- pont.log (fin) ---
|