gump2049 commited on
Commit
ae6460f
·
verified ·
1 Parent(s): 96919c0

Restore four-model latency chart with updated TypeSafe percentiles

Browse files
README.md CHANGED
@@ -70,10 +70,11 @@ The 4B model leads this panel's Browser subset, while 9B's gains come from princ
70
 
71
  *Figure 3. 9B · effort: high · vLLM HTTP API full-response latency · one RTX PRO 6000 Blackwell 96GB GPU. The supplied run snapshot reports 333/333 successful requests and 0% errors. This is full-response latency, not streaming first-token latency.*
72
 
73
- | vLLM HTTP model | P50 | P95 | P99 | Source |
74
  | --- | ---: | ---: | ---: | --- |
75
  | **APUS-OpenJev 9B · high** | **25.58 ms** | **222.36 ms** | **275.58 ms** | Supplied run snapshot |
76
  | **APUS-OpenJev 4B** | **18.00 ms** | **100.06 ms** | **154.32 ms** | Reported P50; ratio-derived P95/P99 |
 
77
 
78
  The 4B P50 uses the updated reported result of **18 ms**. Its P95/P99 apply the supplied cross-session ratios of **45% / 56%** to the corresponding 9B high percentiles, rounded to two decimals. The 9B snapshot reports **333/333 successful requests and 0% errors**; these request counts are not attributed to 4B.
79
 
@@ -90,7 +91,7 @@ The optimized path is the candidate-scoring `/decide` service. The standard Chat
90
  | Round 3 | 305.57 ms | 65/80 · 81.25% |
91
  | **Overall** | **302.65 ms** | **199/240 · 82.92%** |
92
 
93
- These supplied screenshot results report the arithmetic mean, not P50. Updated TypeSafe percentiles were not supplied. The earlier task-accuracy charts retain their original single-run results; this table records the separate three-round evaluation.
94
 
95
  **Network reference from the same Runpod host.** The API evaluation used **240 consecutive calls** across three rounds. Separate network probes reported:
96
 
 
70
 
71
  *Figure 3. 9B · effort: high · vLLM HTTP API full-response latency · one RTX PRO 6000 Blackwell 96GB GPU. The supplied run snapshot reports 333/333 successful requests and 0% errors. This is full-response latency, not streaming first-token latency.*
72
 
73
+ | Serving path | P50 | P95 | P99 | Source |
74
  | --- | ---: | ---: | ---: | --- |
75
  | **APUS-OpenJev 9B · high** | **25.58 ms** | **222.36 ms** | **275.58 ms** | Supplied run snapshot |
76
  | **APUS-OpenJev 4B** | **18.00 ms** | **100.06 ms** | **154.32 ms** | Reported P50; ratio-derived P95/P99 |
77
+ | **Jev / TypeSafe official API** | **280.80 ms** | **376.56 ms** | **830.45 ms** | Supplied pooled 240-request results |
78
 
79
  The 4B P50 uses the updated reported result of **18 ms**. Its P95/P99 apply the supplied cross-session ratios of **45% / 56%** to the corresponding 9B high percentiles, rounded to two decimals. The 9B snapshot reports **333/333 successful requests and 0% errors**; these request counts are not attributed to 4B.
80
 
 
91
  | Round 3 | 305.57 ms | 65/80 · 81.25% |
92
  | **Overall** | **302.65 ms** | **199/240 · 82.92%** |
93
 
94
+ The updated screenshot reports **P50 280.80 ms, P95 376.56 ms and P99 830.45 ms**, computed from all **240 HTTP 200 responses pooled across three rounds**, rather than averaging per-round percentiles. Mean response time remains **302.65 ms**. Timing includes the public-network overhead from Runpod to the official service; 80 questions repeated three times provide a sample-specific tail-latency reference. The earlier task-accuracy charts retain their original single-run results; this table records the separate three-round evaluation.
95
 
96
  **Network reference from the same Runpod host.** The API evaluation used **240 consecutive calls** across three rounds. Separate network probes reported:
97
 
README.zh-CN.md CHANGED
@@ -70,10 +70,11 @@ tags:
70
 
71
  ![9B vLLM决策接口:P50 25.58 ms、P95 222.36 ms、P99 275.58 ms。](assets/latency-vllm.svg)
72
 
73
- | vLLM HTTP模型 | P50 | P95 | P99 | 来源 |
74
  | --- | ---: | ---: | ---: | --- |
75
  | **APUS-OpenJev 9B · high** | **25.58 ms** | **222.36 ms** | **275.58 ms** | 提供的运行截图 |
76
  | **APUS-OpenJev 4B** | **18.00 ms** | **100.06 ms** | **154.32 ms** | P50采用最新提供值;P95/P99按比例换算 |
 
77
 
78
  4B P50采用最新提供的 **18 ms**;P95/P99分别按9B high对应值的 **45% / 56%** 计算,保留两位小数。9B截图中的成功请求为 **333/333、错误率0%**,不将该请求数套用到4B。
79
 
@@ -90,7 +91,7 @@ tags:
90
  | 第 3 轮 | 305.57 ms | 65/80 · 81.25% |
91
  | **三轮总体** | **302.65 ms** | **199/240 · 82.92%** |
92
 
93
- 以上采用提供截图中的算术平均耗时,不是 P50;截图未提供更新的 TypeSafe 分位数。前面的分任务准确率图保留原单轮成绩,此表单独记录三轮复测。
94
 
95
  **同一台 Runpod 的网络参考。** API 评测采用三轮、共 **240 次连续调用**。另行网络探测结果如下:
96
 
 
70
 
71
  ![9B vLLM决策接口:P50 25.58 ms、P95 222.36 ms、P99 275.58 ms。](assets/latency-vllm.svg)
72
 
73
+ | 服务路径 | P50 | P95 | P99 | 来源 |
74
  | --- | ---: | ---: | ---: | --- |
75
  | **APUS-OpenJev 9B · high** | **25.58 ms** | **222.36 ms** | **275.58 ms** | 提供的运行截图 |
76
  | **APUS-OpenJev 4B** | **18.00 ms** | **100.06 ms** | **154.32 ms** | P50采用最新提供值;P95/P99按比例换算 |
77
+ | **Jev / TypeSafe 官方 API** | **280.80 ms** | **376.56 ms** | **830.45 ms** | 提供的240次请求合并统计 |
78
 
79
  4B P50采用最新提供的 **18 ms**;P95/P99分别按9B high对应值的 **45% / 56%** 计算,保留两位小数。9B截图中的成功请求为 **333/333、错误率0%**,不将该请求数套用到4B。
80
 
 
91
  | 第 3 轮 | 305.57 ms | 65/80 · 81.25% |
92
  | **三轮总体** | **302.65 ms** | **199/240 · 82.92%** |
93
 
94
+ 最新截图提供 **P50 280.80 ms、P95 376.56 ms、P99 830.45 ms**,由三轮全部 **240 条 HTTP 200 响应记录合并后重新计算**,不是各轮分位数取平均。平均耗时仍为 **302.65 ms**,包含 Runpod 到官方服务的网络开销;80题重复三次的 P99 仅作为该样本的尾延迟参考。前面的分任务准确率图保留原单轮成绩,此表单独记录三轮复测。
95
 
96
  **同一台 Runpod 的网络参考。** API 评测采用三轮、共 **240 次连续调用**。另行网络探测结果如下:
97
 
TECHNICAL_REPORT.md CHANGED
@@ -198,6 +198,7 @@ The APUS serving reference uses **one RTX PRO 6000 Blackwell 96GB GPU**. The 9B
198
  |---|---:|---:|---:|
199
  | APUS-OpenJev 9B high · vLLM HTTP | 25.58 ms | 222.36 ms | 275.58 ms |
200
  | APUS-OpenJev 4B · vLLM HTTP | 18.00 ms | 100.06 ms | 154.32 ms |
 
201
  | Laya typed · complete-input local pipeline | 8.68 ms | 37.66 ms | — |
202
 
203
  The 9B-5949 high snapshot reports **333/333 successful requests and 0% errors**. An independently archived three-round serial HTTP reference reports median per-round P50/P95/P99 of **25.34 / 222.83 / 276.78 ms**, with **3,120/3,120** successful requests. These two run records remain separate.
@@ -215,7 +216,7 @@ Jev includes public-network, queueing, and service overhead. Laya includes token
215
  | Round 3 | 305.57 ms | 65/80 · 81.25% |
216
  | **Overall** | **302.65 ms** | **199/240 · 82.92%** |
217
 
218
- These supplied screenshot results report the arithmetic mean, not P50. Updated TypeSafe percentiles were not supplied. The earlier task-accuracy charts retain their original single-run results; this table records the separate three-round evaluation.
219
 
220
  **Network reference from the same Runpod host.** The API evaluation used **240 consecutive calls** across three rounds. Separate network probes reported:
221
 
 
198
  |---|---:|---:|---:|
199
  | APUS-OpenJev 9B high · vLLM HTTP | 25.58 ms | 222.36 ms | 275.58 ms |
200
  | APUS-OpenJev 4B · vLLM HTTP | 18.00 ms | 100.06 ms | 154.32 ms |
201
+ | Jev / TypeSafe official API · public HTTP | 280.80 ms | 376.56 ms | 830.45 ms |
202
  | Laya typed · complete-input local pipeline | 8.68 ms | 37.66 ms | — |
203
 
204
  The 9B-5949 high snapshot reports **333/333 successful requests and 0% errors**. An independently archived three-round serial HTTP reference reports median per-round P50/P95/P99 of **25.34 / 222.83 / 276.78 ms**, with **3,120/3,120** successful requests. These two run records remain separate.
 
216
  | Round 3 | 305.57 ms | 65/80 · 81.25% |
217
  | **Overall** | **302.65 ms** | **199/240 · 82.92%** |
218
 
219
+ The updated screenshot reports **P50 280.80 ms, P95 376.56 ms and P99 830.45 ms**, computed from all **240 HTTP 200 responses pooled across three rounds**, rather than averaging per-round percentiles. Mean response time remains **302.65 ms**. Timing includes the public-network overhead from Runpod to the official service; 80 questions repeated three times provide a sample-specific tail-latency reference. The earlier task-accuracy charts retain their original single-run results; this table records the separate three-round evaluation.
220
 
221
  **Network reference from the same Runpod host.** The API evaluation used **240 consecutive calls** across three rounds. Separate network probes reported:
222
 
TECHNICAL_REPORT.pdf CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2a0cd1e5fe4b2a496820437886660dff564fb9209774a1bafb92df8a38e86ee4
3
- size 964744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c9845082fe7a78e5fc92979e830ce74a63fbcb33f42a5452048eaeea1f80e6a
3
+ size 959961
assets/latency-vllm.svg CHANGED
bundle-manifest.json CHANGED
@@ -583,23 +583,23 @@
583
  },
584
  {
585
  "path": "README.md",
586
- "bytes": 18318,
587
- "sha256": "d76dbe15deee7992b1aa77feb67ee370f2c9a28fdbce23dc9fec60ddc56e4d14"
588
  },
589
  {
590
  "path": "README.zh-CN.md",
591
- "bytes": 13684,
592
- "sha256": "1a97275ae1552ad13b0224ef00676b5401b12357e415d397d49d2ad4022bf503"
593
  },
594
  {
595
  "path": "TECHNICAL_REPORT.md",
596
- "bytes": 28037,
597
- "sha256": "474867a3f7ebefbc6db8b77785aa7b37dc0881245c1b0241c8186b61befec995"
598
  },
599
  {
600
  "path": "TECHNICAL_REPORT.pdf",
601
- "bytes": 964744,
602
- "sha256": "2a0cd1e5fe4b2a496820437886660dff564fb9209774a1bafb92df8a38e86ee4"
603
  },
604
  {
605
  "path": "assets/accuracy.svg",
@@ -623,8 +623,8 @@
623
  },
624
  {
625
  "path": "assets/latency-vllm.svg",
626
- "bytes": 9526,
627
- "sha256": "a96b1adeceb7a21b52a651a19f5da72fabcf14f7a8ca2338b141f3eaf8adc9c2"
628
  },
629
  {
630
  "path": "assets/latency.svg",
@@ -658,8 +658,8 @@
658
  },
659
  {
660
  "path": "latency-vllm-data.json",
661
- "bytes": 3144,
662
- "sha256": "12107b27f37e8f2bd3c3cd124ba653d6a66cc4e476ab3a1d1115014a337ea0aa"
663
  },
664
  {
665
  "path": "subset-chart-data.json",
 
583
  },
584
  {
585
  "path": "README.md",
586
+ "bytes": 18726,
587
+ "sha256": "1ed50a42e5899615fea3d22f2447858d71a39c516ac863c0818cd5795882763a"
588
  },
589
  {
590
  "path": "README.zh-CN.md",
591
+ "bytes": 14012,
592
+ "sha256": "8fc192b2d8d46b50be17cb985d01ab3c742c41228638126c8e6827f8dd158298"
593
  },
594
  {
595
  "path": "TECHNICAL_REPORT.md",
596
+ "bytes": 28409,
597
+ "sha256": "df3415c74deb51038fef662f8658c499d5d07f097d30efd7f1c8dd93ab664cbf"
598
  },
599
  {
600
  "path": "TECHNICAL_REPORT.pdf",
601
+ "bytes": 959961,
602
+ "sha256": "0c9845082fe7a78e5fc92979e830ce74a63fbcb33f42a5452048eaeea1f80e6a"
603
  },
604
  {
605
  "path": "assets/accuracy.svg",
 
623
  },
624
  {
625
  "path": "assets/latency-vllm.svg",
626
+ "bytes": 8311,
627
+ "sha256": "984b50eb9745d7b02cdc0625a071713c88a5461405905cab0c6282657609bdca"
628
  },
629
  {
630
  "path": "assets/latency.svg",
 
658
  },
659
  {
660
  "path": "latency-vllm-data.json",
661
+ "bytes": 3456,
662
+ "sha256": "3165a4510d2d294934b0374d1f6e9a53303f5fa734daf662eb8a3aa69581fab0"
663
  },
664
  {
665
  "path": "subset-chart-data.json",
latency-vllm-data.json CHANGED
@@ -19,7 +19,7 @@
19
  },
20
  "typesafe_three_rounds": {
21
  "source": "User-provided TypeSafe three-round screenshot, 2026-09-21",
22
- "timing": "Official public API HTTP response; arithmetic mean, not P50",
23
  "raw_requests_independently_verified": false,
24
  "rounds": [
25
  {
@@ -45,9 +45,11 @@
45
  "overall_correct": 199,
46
  "overall_requests": 240,
47
  "overall_accuracy_percent": 82.92,
48
- "p50_ms": null,
49
- "p95_ms": null,
50
- "p99_ms": null,
 
 
51
  "note": "Historical single-run accuracy charts retain their original run; this is a separate repeated evaluation."
52
  },
53
  "provided_snapshot": {
@@ -96,6 +98,12 @@
96
  }
97
  },
98
  "historical_rows": [
 
 
 
 
 
 
99
  {
100
  "name": "Laya typed / full input",
101
  "scope": "Local system_one / three-run median",
 
19
  },
20
  "typesafe_three_rounds": {
21
  "source": "User-provided TypeSafe three-round screenshot, 2026-09-21",
22
+ "timing": "Official public API full HTTP response; pooled percentiles across all 240 requests",
23
  "raw_requests_independently_verified": false,
24
  "rounds": [
25
  {
 
45
  "overall_correct": 199,
46
  "overall_requests": 240,
47
  "overall_accuracy_percent": 82.92,
48
+ "p50_ms": 280.8,
49
+ "p95_ms": 376.56,
50
+ "p99_ms": 830.45,
51
+ "aggregation": "Percentiles recomputed on pooled 240 records, not averages of per-round percentiles",
52
+ "http_200_responses": 240,
53
  "note": "Historical single-run accuracy charts retain their original run; this is a separate repeated evaluation."
54
  },
55
  "provided_snapshot": {
 
98
  }
99
  },
100
  "historical_rows": [
101
+ {
102
+ "name": "Jev official API",
103
+ "scope": "TypeSafe HTTP / 240 pooled requests",
104
+ "p50_ms": 280.8,
105
+ "p95_ms": 376.56
106
+ },
107
  {
108
  "name": "Laya typed / full input",
109
  "scope": "Local system_one / three-run median",