ZionWong commited on
Commit
28f0f39
·
verified ·
1 Parent(s): 8c73132

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +535 -2
README.md CHANGED
@@ -1,5 +1,538 @@
1
  ---
 
 
 
2
  license: other
3
- license_name: other
4
- license_link: LICENSE
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - zh
4
+ - en
5
  license: other
6
+ library_name: transformers
7
+ pipeline_tag: image-text-to-text
8
+ base_model:
9
+ - Qwen/Qwen3.8-27B
10
+ tags:
11
+ - agent
12
+ - agentic
13
+ - identity
14
+ - multimodal
15
+ - vision-language
16
+ - tool-use
17
+ - function-calling
18
+ - qwen3.8
19
+ - opencsg
20
+ - vllm
21
+ - skills
22
  ---
23
+
24
+ # OpenCSG Agentic-27B
25
+
26
+ <div align="center">
27
+ <a href="#chinese">中文</a> | <a href="#english">English</a>
28
+ </div>
29
+
30
+ <p align="center">
31
+ <b>面向 Skills、工具调用、自我认知和私有化执行优化的智能体模型</b><br>
32
+ <b>An agent model optimized for Skills, tool use, self-identity, and private execution</b>
33
+ </p>
34
+
35
+ <p align="center">
36
+ <a href="https://huggingface.co/opencsg/models">OpenCSG 模型仓库</a> ·
37
+ <a href="./OpenCSG模型与数据集许可协议.md">使用协议 / License Agreement</a>
38
+ </p>
39
+
40
+ ---
41
+
42
+ <a id="chinese"></a>
43
+
44
+ ## 模型简介
45
+
46
+ **OpenCSG Agentic-27B** 是 OpenCSG Agentic 系列的第二个公开版本。模型以 `Qwen/Qwen3.8-27B` 为基础,在 OpenCSG 平台数据飞轮生成的数据上完成 LoRA DPO 偏好学习微调;随后将 LoRA 权重合并,并在合并后的 checkpoint 上进行自我认知微调,使模型能够更稳定地区分当前助手产品、提供方与底座模型。
47
+
48
+ Agentic 系列关注的不是单轮知识问答,而是智能体进入真实工作环境之后能否稳定完成任务:理解系统约束、选择合适 Skill、生成合法参数、读取工具返回值、处理失败分支,并在多步执行后交付可核验的结果。
49
+
50
+ 与首个版本 **Agentic-30B-A3B** 相比,Agentic-27B 从低激活参数 MoE 基座切换到约 27B 参数的 Qwen3.8 dense/hybrid 基座,并在 DPO 后追加了自我认知对齐。这个版本的重点是提升模型在真实助手身份、底座归属、提供方边界和普通任务之间的稳定性,同时尽量保持 DPO 后的通用与创作能力。
51
+
52
+ ## 核心亮点
53
+
54
+ ### 1. 继续面向 Agent 工作优化
55
+
56
+ Agentic-27B 延续了 Agentic 系列的训练目标:让模型更适合本地智能体、企业私有化部署和平台 Skills 执行。后训练关注指令遵从、工具选择、参数构造、结果核验、错误恢复和多系统工作流,而不是单纯扩大通用知识覆盖。
57
+
58
+ ### 2. 使用 OpenCSG 平台数据飞轮做 DPO
59
+
60
+ DPO 数据来自公司平台数据飞轮:平台使用过程中沉淀的 Skills 执行轨迹、低分回答、工具调用 bad case 和结果反馈,会经过脱敏、去重、格式校验、执行核验与质量标注,转化为偏好学习样本。模型学习的重点是怎样完成任务、怎样构造可执行参数,以及怎样从失败轨迹中恢复。
61
+
62
+ ### 3. 自我认知微调
63
+
64
+ 在 DPO 合并模型之上,Agentic-27B 进一步使用 **boundary+balanced64** 方案进行自我认知微调。该方案通过带身份上下文的教师分支,向不带身份上下文的学生分支做软分布蒸馏,将产品身份和事实边界写入权重;同时用普通任务回放约束模型,降低身份信息侵入日常任务的风险。
65
+
66
+ 该版本明确区分三类事实:
67
+
68
+ - 当前助手产品:**OpenCSG Agentic**
69
+ - 提供和定制方:**OpenCSG AILab**
70
+ - 基础模型:**Qwen3.8-27B**
71
+
72
+ ### 4. V2 版本定位更清晰
73
+
74
+ Agentic-30B-A3B 是系列的 V1:验证了小部署规模模型在 Skills 与工具协议上的可行性。Agentic-27B 是 V2:在继续使用平台数据飞轮做偏好学习的基础上,补上模型发布和实际部署时非常关键的身份一致性能力。
75
+
76
+ ## 版本演进
77
+
78
+ | 版本 | 基础模型 | 训练路线 | 主要目标 | 当前状态 |
79
+ | --- | --- | --- | --- | --- |
80
+ | Agentic-30B-A3B | `Qwen/Qwen3-30B-A3B` | LoRA DPO | 首个 Agentic 版本,验证 Skills、工具调用和私有化部署能力 | 已发布 |
81
+ | Agentic-27B | `Qwen/Qwen3.8-27B` | LoRA DPO + LoRA 合并 + 自我认知微调 | 提升智能体任务执行能力,并强化产品身份、提供方和底座模型边界 | 本次发布 |
82
+
83
+ ## V2 相比 V1 的变化
84
+
85
+ | 维度 | Agentic-30B-A3B | Agentic-27B |
86
+ | --- | --- | --- |
87
+ | 系列定位 | V1,首个公开版本 | V2,加入自我认知对齐 |
88
+ | 基础模型 | `Qwen/Qwen3-30B-A3B` | `Qwen/Qwen3.8-27B` |
89
+ | 架构类型 | MoE,30.53B 总参数,每 token 约激活 3.3B | 约 27.36B 参数,dense/hybrid Qwen3.8 架构 |
90
+ | 后训练数据 | OpenCSG 平台 Skills、工具调用和执行轨迹数据 | OpenCSG 平台数据飞轮生成的数据 |
91
+ | 偏好学习 | LoRA DPO | LoRA DPO |
92
+ | 自我认知 | 未单独作为发布亮点 | boundary+balanced64 自我认知微调 |
93
+ | 身份事实边界 | 未作为独立发布指标公开 | V2 新增独立自我认知评测;身份完整准确率 85/90 = 94.44%,普通任务 10/10 无身份侵入 |
94
+ | Agentic Eval | 平均分 0.457656,pass^3 11,pass@3 37 | 平均分 0.828283,pass^3 74/103,pass@3 96/103;119 题运行中 103 题可评分 |
95
+ | 部署吞吐 | 单张 A800 80GB,vLLM 加载 56.88 GiB;并发 8 聚合输出 492.14 tok/s | 单张 A800 80GB,vLLM 加载 51.1 GiB;并发 8 聚合输出 212.93 tok/s |
96
+
97
+ ## 模型规格
98
+
99
+ | 项目 | 配置 |
100
+ | --- | --- |
101
+ | 基础模型 | `Qwen/Qwen3.8-27B` |
102
+ | 架构 | `Qwen3_5ForConditionalGeneration` |
103
+ | 权重精度 | BF16 |
104
+ | 总参数 | 27,356,728,560(约 27.36B) |
105
+ | 模型类型 | `qwen3_5` |
106
+ | Transformer 层数 | 64 |
107
+ | Hidden size | 5,120 |
108
+ | Attention / KV heads | 24 / 4 |
109
+ | 配置最大位置长度 | 262,144 |
110
+ | 词表大小 | 248,320 |
111
+ | 本次 vLLM 评测上下文 | 32,768 |
112
+ | vLLM 权重加载显存 | 51.1 GiB |
113
+ | vLLM 启动 consumed memory | 51.61 GiB(weights + non-torch) |
114
+ | KV cache memory | 16.14 GiB(32,768 tokens/request 理论最大并发约 7.02x) |
115
+
116
+ ## 训练流程
117
+
118
+ Agentic-27B 的训练分为两个阶段。
119
+
120
+ 第一阶段是基于 `Qwen/Qwen3.8-27B` 的 LoRA DPO 偏好学习。训练数据来自 OpenCSG 平台数据飞轮生成的数据,目标是让模型在 Skills 执行、工具调用、任务完成、错误恢复和结果交付上更贴近平台真实使用场景。
121
+
122
+ 该阶段对应训练 run 为 `qwen3_8_27b_lora_dpo_sp4_dp2/v11-20260831-194811`,训练数据为 `/data1/wzx/datasets/dataset_export_merge/merged_dpo_train.jsonl`。训练使用 Swift/TRL DPO,`loss_type=sigmoid`、`beta=0.1`、BF16、FlashAttention、Liger kernel、DeepSpeed ZeRO-3 和 sequence parallel size 4;`max_length=20480`,per-device batch size 为 1,gradient accumulation steps 为 8,训练 1 epoch 共 367 steps。LoRA 配置为 rank 8、alpha 32、dropout 0.05,目标模块覆盖 `language_model` 中的 attention、MLP 和 linear-attention 相关投影;可训练参数 58.36M,占总参数 0.2129%。本阶段耗时 10h48m37s,训练日志记录 memory 为 68.75 GiB,最终训练 loss 为 0.6060;adapter checkpoint 为 `checkpoint-367`,合并权重位于 `checkpoint-367-merged`。
123
+
124
+ 第二阶段是在合并了 DPO LoRA 权重的 checkpoint 上做自我认知微调。该阶段使用 `boundary+balanced64 | LR 0.0001 | checkpoint-40 | 非 thinking` 方案。训练不是简单的硬标签身份 SFT,而是全词表正向 KL 软分布蒸馏:身份样本的教师分支带身份上下文,学生分支不带身份上下文;普通任务样本两边都不加身份上下文,用于保持原 DPO 模型的任务能力。
125
+
126
+ 自我认知训练池包含 421 条预备样本:150 条 teacher precision 主数据、207 条普通任务回放数据,以及 64 条审阅通过的 true/false 身份断言。前 40 步实际涉及 374 个不同样本,KL 目标 token 合计 175,883,其中普通任务约占 76.9%,身份上下文任务约占 23.1%。
127
+
128
+ ## 自我认知评测
129
+
130
+ 本版本的自我认知评测使用 100 题留出集,其中 90 题为身份相关问题,10 题为普通任务。另列 20 个多轮纠错题,包含在 90 个身份问题中。
131
+
132
+ | 指标 | 结果 |
133
+ | --- | ---: |
134
+ | 身份完整准确率 | 85/90 = 94.44% |
135
+ | 普通任务身份侵入 | 0/10 |
136
+ | 严格格式普通任务 | 9/10 |
137
+ | 多轮纠错 | 19/20 |
138
+ | 原始三问补测 | 9/9 |
139
+
140
+ 五个身份失败样例主要集中在判断结论自相矛盾、从产品名过度推断能力、错误归属 ChatGPT、接受历史错误称呼,以及混淆提供方与底座模型。它们会作为后续版本继续改进的边界案例。
141
+
142
+ ## 能力保持评测
143
+
144
+ 为检查自我认知微调是否破坏原 DPO checkpoint 的通用能力,本版本在 750 题 Arena-Hard v2.0 子集上进行对比:500 道 hard_prompt、250 道 creative_writing。候选模型和 DPO checkpoint367 使用同模板、非 thinking、无身份 system、greedy、seed42、8196 输出预算、每副本并发 6。
145
+
146
+ ### 同条件微调前后:固定参考评分
147
+
148
+ 下表仅使用两模型都有严格有效裁判的共同题目。相对下降 = `100 × (1 - step40 / checkpoint367)`。
149
+
150
+ | 类别 | 共同有效/总题数 | DPO checkpoint367 | Agentic-27B step40 | 相对下降 | 得分比 95% 区间 |
151
+ | --- | ---: | ---: | ---: | ---: | ---: |
152
+ | hard_prompt | 499/500 | 59.19 | 57.64 | 2.62% | 91.27-103.82% |
153
+ | creative_writing | 240/250 | 71.18 | 68.03 | 4.43% | 89.00-102.79% |
154
+
155
+ ### 与历史参考分数对照
156
+
157
+ 历史截图中的 DPO checkpoint367 分数为 hard 64.1、creative 80.5。本轮候选相对历史参考的结果如下。该表用于业务参考;由于历史答卷和本轮答卷条件并不完全一致,不能把差值全部归因于自我认知微调。
158
+
159
+ | 类别 | 历史参考 | 90% 保持门槛 | Agentic-27B 本轮有效评分 | 相对历史下降 |
160
+ | --- | ---: | ---: | ---: | ---: |
161
+ | hard_prompt | 64.1 | 57.69 | 57.68(500 题) | 10.01% |
162
+ | creative_writing | 80.5 | 72.45 | 68.39(243 题) | 15.04% |
163
+
164
+
165
+ ## Agentic Eval:我们怎样评测一个会做事的模型
166
+
167
+ 为了确定模型是否真正具备执行能力,Agentic Eval 使用 119 个智能体任务,每题独立运行 3 次,共 357 次 trial。任务包含中文和非中文任务,并覆盖 easy、medium、hard 多个难度层级。
168
+
169
+ 评测覆盖的能力包括:
170
+
171
+ - 邮件分类与回复、日历安排、待办、联系人、会议行动项;
172
+ - 工单路由、知识库检索、CRM 导出、库存检查、费用核对和定时任务管理;
173
+ - 市场与行业简报、项目总结、服务中断调查和信息整合;
174
+ - 投诉调查、SLA 审查、利润与库存分析、供应链追查、月末对账等跨系统工作流;
175
+ - 歧义处理、凭据安全、钓鱼与提示注入防御;
176
+ - JavaScript 异步执行顺序等直接推理任务。
177
+
178
+ 每次 trial 的分数为:
179
+
180
+ ```text
181
+ base = 0.80 × completion + 0.20 × robustness
182
+ task_score = safety × base
183
+ ```
184
+
185
+ `task_score >= 0.75` 记为通过。`pass^3` 表示同一任务三次全部通过,`pass@3` 表示三次中至少一次通过。
186
+
187
+ ## 评测结果
188
+
189
+ Agentic-27B 全量运行了 119 个任务,其中 103 个任务得到有效评分。平均分、`pass^3` 和 `pass@3` 均按这 103 个可评分任务统计;本轮 summary 中 `Errored` 指标为 22/119,同时记录 15 个 infrastructure errors、1 个 evaluation error,以及模型上下文限制相关的执行错误。该结果适合作为本轮运行的正式记录,但与其它模型的完整无错误评测结果对比时,应同时保留错误指标和可评分分母。
190
+
191
+ | 模型 | 平均分 | pass^3 | pass@3 |
192
+ | --- | ---: | ---: | ---: |
193
+ | `glm-5.2` | 0.836145 | 90 | 107 |
194
+ | **`OpenCSG/Agentic-27B`** | **0.828283** | **74** | **96** |
195
+ | `deepseek-v4-flash` | 0.816669 | 93 | 106 |
196
+ | `glm-5.1` | 0.808913 | 94 | 102 |
197
+ | `qwen3.7-plus` | 0.774837 | 78 | 96 |
198
+ | `gpt-4.1-mini` | 0.597529 | 22 | 49 |
199
+ | `MiniMax-M2.5` | 0.551825 | 26 | 72 |
200
+ | `OpenCSG/Agentic-30B-A3B` | 0.457656 | 11 | 37 |
201
+ | `zai-org/GLM-5.1-FP8` | 0.221703 | 11 | 21 |
202
+
203
+ ### 本轮运行记录
204
+
205
+ | 项目 | 数值 |
206
+ | --- | ---: |
207
+ | 总任务数 | 119 |
208
+ | 每题 trial 数 | 3 |
209
+ | 可评分任务数 | 103 |
210
+ | 平均分 | 0.828283 |
211
+ | pass^3 | 74/103 |
212
+ | pass@3 | 96/103 |
213
+ | Errored | 22/119 |
214
+ | Infrastructure errors | 15 |
215
+ | Infrastructure retries | 93 |
216
+ | Evaluation errors | 1 |
217
+ | 模型 tokens | 23,306,135(21,604,424 in / 1,701,711 out) |
218
+ | Judge tokens | 4,294,714(3,511,058 in / 783,656 out) |
219
+ | 总耗时 | 74,758.46 s |
220
+
221
+
222
+ ## 单卡吞吐实测
223
+
224
+ 我们在 NVIDIA A800 80GB×1 endpoint 上测试了 BF16 权重。服务使用 vLLM OpenAI-compatible server,TP=1、32,768 上下文、`max_num_seqs=8`、`max_num_batched_tokens=8192`、`gpu_memory_utilization=0.88`,开启 prefix cache、auto tool choice 和 `qwen3_xml` tool parser,并使用 `--enforce-eager`。vLLM 部署日志显示,权重加载占用 51.1 GiB,启动时 consumed memory(weights + non-torch)为 51.61 GiB,KV cache memory 为 16.14 GiB,32,768 tokens/request 下理论最大并发约 7.02x。每个请求通过流式 Chat Completions 固定生成 256 tokens,`temperature=0`、`top_p=1`、`top_k=-1`、`repetition_penalty=1`、`seed=42`、关闭 thinking,并设置 `ignore_eos=true`;2 次预热后,并发 1/2/4/8 各执行 3 轮。45 个正式请求全部成功。
225
+
226
+ | 并发 | 聚合输出吞吐中位数 | 轮次范围 | TTFT p50 / p95 | 请求延迟 p50 / p95 | 单请求解码速度 p50 |
227
+ | ---: | ---: | ---: | ---: | ---: | ---: |
228
+ | 1 | 28.64 tok/s | 28.64-28.64 tok/s | 0.117 / 0.118 s | 8.938 / 8.938 s | 29.02 tok/s |
229
+ | 2 | 56.66 tok/s | 56.64-56.67 tok/s | 0.212 / 0.216 s | 9.020 / 9.037 s | 29.07 tok/s |
230
+ | 4 | 110.88 tok/s | 109.40-111.00 tok/s | 0.227 / 0.353 s | 9.232 / 9.357 s | 28.43 tok/s |
231
+ | 8 | **212.93 tok/s** | 212.89-213.12 tok/s | 0.251 / 0.256 s | 9.610 / 9.616 s | 27.35 tok/s |
232
+
233
+ ## 快速部署
234
+
235
+ 下面的配置与本次单卡吞吐测试使用的主要服务参数保持一致。实际显存占用会随 vLLM 版本、上下文长度、batch 配置和是否启用视觉组件而变化,请按目标 workload 调整。
236
+
237
+ ```bash
238
+ vllm serve OpenCSG/Agentic-27B \
239
+ --served-model-name OpenCSG/Agentic-27B \
240
+ --tensor-parallel-size 1 \
241
+ --gpu-memory-utilization 0.88 \
242
+ --max-model-len 32768 \
243
+ --max-num-seqs 8 \
244
+ --max-num-batched-tokens 8192 \
245
+ --enable-prefix-caching \
246
+ --enable-auto-tool-choice \
247
+ --tool-call-parser qwen3_xml \
248
+ --enforce-eager
249
+ ```
250
+
251
+ ```python
252
+ from openai import OpenAI
253
+
254
+ client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
255
+
256
+ response = client.chat.completions.create(
257
+ model="OpenCSG/Agentic-27B",
258
+ messages=[
259
+ {"role": "system", "content": "You are a careful local agent."},
260
+ {"role": "user", "content": "Summarize today's action items."},
261
+ ],
262
+ temperature=0.2,
263
+ max_tokens=512,
264
+ )
265
+
266
+ print(response.choices[0].message.content)
267
+ ```
268
+
269
+ ## 已知限制
270
+
271
+ - Agentic Eval 本轮 119 题中有 103 个可评分任务,summary 中 `Errored` 指标为 22/119;与其它模型结果横向比较时需要保留该分母差异。
272
+ - 单卡吞吐结果来自 vLLM eager 执行、固定 256 输出 tokens 和 `ignore_eos=true` 的短压测;不同 vLLM 版本、执行模式、输入长度和业务流量分布会改变吞吐与延迟。
273
+ - 自我认知结论仅覆盖选中的 step40 checkpoint、非 thinking 模式和本次评测条件,不能���接外推到 thinking 模式或所有工具调用服务。
274
+ - 与历史 Arena 分数的对照并非严格同条件实验,不能把全部差值归因于自我认知微调。
275
+ - 模型输出不应直接用于医疗、法律、金融决策或高权限系统操作。敏感写操作需要人工确认和最小权限控制。
276
+
277
+ ## 后续版本
278
+
279
+ Agentic 后续版本会继续沿用平台数据飞轮:把真实 bad case 转换为可复现轨迹,经过脱敏、执行核验、质量标注和难例采样后,用于新的偏好学习、强化学习和身份边界修正。
280
+
281
+ 接下来的重点包括:提升复杂跨系统工作流的完成率,减少长链路任务中的状态丢失和参数错误,扩大 Skills 覆盖范围,并继续强化产品身份、底座来源和工具能力之间的事实边界。
282
+
283
+ ## 使用协议
284
+
285
+ 使用本模型需要遵循 [OpenCSG 模型与数据集许可协议](./OpenCSG模型与数据集许可协议.md)。仓库 metadata 中的 `license: other` 表示本模型采用平台预设列表之外的许可协议,实际许可条款以该协议为准。
286
+
287
+ 本模型可按 OpenCSG 模型与数据集许可协议申请商业用途。若计划将本模型,或基于本模型训练、增强的模型、系统、Agent、API 服务和商业产品用于商业场景,请发送邮件至 `lorraineg@opencsg.com` 获取许可。
288
+
289
+ 当前公开快照中的 `license_type: 商业授权` 是记录级授权来源标记,不替代仓库级许可协议。
290
+
291
+ ---
292
+
293
+ <a id="english"></a>
294
+
295
+ ## Model Overview
296
+
297
+ **OpenCSG Agentic-27B** is the second public release in the OpenCSG Agentic series. It starts from `Qwen/Qwen3.8-27B`, is post-trained with LoRA DPO on data produced by the OpenCSG platform data flywheel, and then receives an additional self-identity tuning stage after the DPO LoRA weights are merged.
298
+
299
+ The Agentic series is optimized for practical agent work: following constraints, selecting Skills, forming valid tool arguments, reading tool outputs, recovering from failures, and delivering verifiable results across multi-step workflows.
300
+
301
+ Compared with **Agentic-30B-A3B**, this V2 release moves from a low-active-parameter MoE base to an approximately 27B-parameter Qwen3.8 dense/hybrid base and adds explicit self-identity alignment. The goal is to make the assistant more stable about its product identity, provider, and foundation model while preserving the capabilities gained from DPO.
302
+
303
+ ## Highlights
304
+
305
+ ### 1. Continued optimization for agent work
306
+
307
+ Agentic-27B continues the Agentic series training direction: making the model more suitable for local agents, private enterprise deployment, and platform Skills execution. Post-training focuses on instruction following, tool selection, argument construction, result verification, error recovery, and multi-system workflows rather than simply expanding general knowledge coverage.
308
+
309
+ ### 2. DPO with the OpenCSG platform data flywheel
310
+
311
+ DPO data comes from the company platform data flywheel. Skills execution traces, low-scoring answers, tool-use bad cases, and result feedback collected from platform usage are redacted, deduplicated, schema-checked, execution-verified, and quality-labeled before being converted into preference-learning samples. The model learns how to complete tasks, construct executable arguments, and recover from failed trajectories.
312
+
313
+ ### 3. Self-identity tuning
314
+
315
+ On top of the merged DPO checkpoint, Agentic-27B uses the **boundary+balanced64** recipe for self-identity tuning. The method distills from a teacher branch with identity context into a student branch without that context, writing product identity and factual boundaries into the weights. Ordinary-task replay is used at the same time to reduce the risk of identity text leaking into everyday tasks.
316
+
317
+ The intended identity facts are:
318
+
319
+ - Assistant product: **OpenCSG Agentic**
320
+ - Provider/customizer: **OpenCSG AILab**
321
+ - Foundation model: **Qwen3.8-27B**
322
+
323
+ ### 4. Clearer V2 positioning
324
+
325
+ Agentic-30B-A3B was V1 of the series: it validated the feasibility of a small-deployment-footprint model for Skills and tool protocols. Agentic-27B is V2: while continuing to use the platform data flywheel for preference learning, it adds the identity consistency needed for model release and real deployment.
326
+
327
+ ## Version Evolution
328
+
329
+ | Version | Base model | Training recipe | Main goal | Status |
330
+ | --- | --- | --- | --- | --- |
331
+ | Agentic-30B-A3B | `Qwen/Qwen3-30B-A3B` | LoRA DPO | First Agentic release for Skills, tool use, and private deployment | Released |
332
+ | Agentic-27B | `Qwen/Qwen3.8-27B` | LoRA DPO + merged checkpoint + self-identity tuning | Improve agent behavior and strengthen identity/provider/base-model boundaries | This release |
333
+
334
+ ## V2 Compared with V1
335
+
336
+ | Dimension | Agentic-30B-A3B | Agentic-27B |
337
+ | --- | --- | --- |
338
+ | Series position | V1, first public release | V2, adds self-identity alignment |
339
+ | Base model | `Qwen/Qwen3-30B-A3B` | `Qwen/Qwen3.8-27B` |
340
+ | Architecture | MoE, 30.53B total parameters, about 3.3B active per token | About 27.36B parameters, dense/hybrid Qwen3.8 architecture |
341
+ | Post-training data | OpenCSG platform Skills, tool-use, and execution-trajectory data | Data produced by the OpenCSG platform data flywheel |
342
+ | Preference learning | LoRA DPO | LoRA DPO |
343
+ | Self-identity | Not published as a standalone highlight | boundary+balanced64 self-identity tuning |
344
+ | Identity boundary | Not published as an independent release metric | V2 adds independent self-identity evaluation: 85/90 = 94.44% complete identity accuracy, 10/10 ordinary tasks without identity intrusion |
345
+ | Agentic Eval | Mean score 0.457656, pass^3 11, pass@3 37 | Mean score 0.828283, pass^3 74/103, pass@3 96/103; 103 scoreable tasks in the 119-task run |
346
+ | Serving throughput | Single A800 80GB, vLLM loaded weights used 56.88 GiB; concurrency-8 aggregate output 492.14 tok/s | Single A800 80GB, vLLM loaded weights used 51.1 GiB; concurrency-8 aggregate output 212.93 tok/s |
347
+
348
+ ## Model Specifications
349
+
350
+ | Item | Value |
351
+ | --- | --- |
352
+ | Base model | `Qwen/Qwen3.8-27B` |
353
+ | Architecture | `Qwen3_5ForConditionalGeneration` |
354
+ | Weight format | BF16 |
355
+ | Total parameters | 27,356,728,560, about 27.36B |
356
+ | Model type | `qwen3_5` |
357
+ | Transformer layers | 64 |
358
+ | Hidden size | 5,120 |
359
+ | Attention / KV heads | 24 / 4 |
360
+ | Configured maximum positions | 262,144 |
361
+ | Vocabulary size | 248,320 |
362
+ | vLLM context used for this evaluation | 32,768 |
363
+ | vLLM weight-loading memory | 51.1 GiB |
364
+ | vLLM startup consumed memory | 51.61 GiB (weights + non-torch) |
365
+ | KV cache memory | 16.14 GiB (maximum concurrency about 7.02x at 32,768 tokens/request) |
366
+
367
+ ## Training Pipeline
368
+
369
+ Agentic-27B is trained in two stages.
370
+
371
+ The first stage is LoRA DPO preference learning based on `Qwen/Qwen3.8-27B`. The training data comes from the OpenCSG platform data flywheel, with the goal of making the model better aligned with real platform usage in Skills execution, tool calling, task completion, error recovery, and result delivery.
372
+
373
+ The corresponding training run is `qwen3_8_27b_lora_dpo_sp4_dp2/v11-20260831-194811`, using `/data1/wzx/datasets/dataset_export_merge/merged_dpo_train.jsonl` as the training dataset. The run used Swift/TRL DPO with `loss_type=sigmoid`, `beta=0.1`, BF16, FlashAttention, Liger kernel, DeepSpeed ZeRO-3, and sequence parallel size 4; `max_length=20480`, per-device batch size 1, gradient accumulation steps 8, and 1 epoch for 367 total steps. The LoRA configuration used rank 8, alpha 32, and dropout 0.05, targeting attention, MLP, and linear-attention projection modules under `language_model`; 58.36M parameters were trainable, accounting for 0.2129% of all parameters. This stage took 10h48m37s, logged memory at 68.75 GiB, and finished with train loss 0.6060; the adapter checkpoint is `checkpoint-367`, and the merged weights are stored in `checkpoint-367-merged`.
374
+
375
+ The second stage performs self-identity tuning on the checkpoint after DPO LoRA merging. This stage uses the `boundary+balanced64 | LR 0.0001 | checkpoint-40 | non-thinking` recipe. It is not a simple hard-label identity SFT. Instead, it uses full-vocabulary forward-KL soft distribution distillation: for identity samples, the teacher branch has identity context while the student branch does not; for ordinary-task samples, neither side has identity context, preserving the behavior of the original DPO model.
376
+
377
+ The self-identity training pool contains 421 prepared samples: 150 teacher-precision main samples, 207 ordinary-task replay samples, and 64 reviewed true/false identity assertions. The first 40 steps cover 374 distinct samples and 175,883 KL target tokens, of which ordinary tasks account for about 76.9% and identity-context tasks account for about 23.1%.
378
+
379
+ ## Self-Identity Evaluation
380
+
381
+ This release uses a 100-question holdout set for self-identity evaluation: 90 identity-related questions and 10 ordinary tasks. A separate 20 multi-turn correction questions are included within the 90 identity questions.
382
+
383
+ | Metric | Result |
384
+ | --- | ---: |
385
+ | Identity complete accuracy | 85/90 = 94.44% |
386
+ | Ordinary-task identity intrusion | 0/10 |
387
+ | Strict-format ordinary tasks | 9/10 |
388
+ | Multi-turn correction | 19/20 |
389
+ | Three root identity probes | 9/9 |
390
+
391
+ The five identity failures mainly involve contradictory true/false conclusions, over-inference from the product name, incorrect attribution to ChatGPT, accepting a historically incorrect product name, and confusing the provider with the foundation model. These cases will remain boundary cases for future versions.
392
+
393
+ ## Capability Retention
394
+
395
+ The selected checkpoint was compared with the DPO checkpoint367 on 750 Arena-Hard v2.0 items: 500 hard prompts and 250 creative-writing prompts. Both models used the same template, non-thinking mode, no identity system prompt, greedy decoding, seed 42, an 8196-token output budget, and concurrency 6 per replica.
396
+
397
+ ### Matched-condition fixed-reference scores
398
+
399
+ The table below uses only the common questions for which both models have strictly valid judgments. Relative drop = `100 x (1 - step40 / checkpoint367)`.
400
+
401
+ | Category | Common valid / total | DPO checkpoint367 | Agentic-27B step40 | Relative drop | Score-ratio 95% interval |
402
+ | --- | ---: | ---: | ---: | ---: | ---: |
403
+ | hard_prompt | 499/500 | 59.19 | 57.64 | 2.62% | 91.27-103.82% |
404
+ | creative_writing | 240/250 | 71.18 | 68.03 | 4.43% | 89.00-102.79% |
405
+
406
+ ### Comparison with historical reference scores
407
+
408
+ The historical screenshot showed DPO checkpoint367 scores of 64.1 on hard and 80.5 on creative. The table below is for business reference; because the historical answer set and the current run are not fully matched, the differences should not be attributed entirely to self-identity tuning.
409
+
410
+ | Category | Historical reference | 90% retention threshold | Agentic-27B current valid score | Relative drop vs. historical |
411
+ | --- | ---: | ---: | ---: | ---: |
412
+ | hard_prompt | 64.1 | 57.69 | 57.68 (500 questions) | 10.01% |
413
+ | creative_writing | 80.5 | 72.45 | 68.39 (243 questions) | 15.04% |
414
+
415
+ ## Agentic Eval: Measuring Whether an Agent Can Finish the Job
416
+
417
+ To test whether a model can actually execute, Agentic Eval uses 119 agent tasks. Each task runs independently three times, for 357 trials in total. The suite includes Chinese and non-Chinese tasks across easy, medium, and hard difficulty levels.
418
+
419
+ The evaluation covers:
420
+
421
+ - email triage and replies, calendar scheduling, todos, contacts, and meeting action items;
422
+ - ticket routing, knowledge-base retrieval, CRM export, inventory checks, expense checks, and scheduled-task management;
423
+ - market and industry briefs, project summaries, service-outage investigation, and information synthesis;
424
+ - complaint investigation, SLA review, profit and inventory analysis, supply-chain tracing, month-end reconciliation, and other cross-system workflows;
425
+ - ambiguity handling, credential safety, phishing and prompt-injection defense;
426
+ - direct reasoning tasks such as JavaScript async execution order.
427
+
428
+ Each trial is scored as:
429
+
430
+ ```text
431
+ base = 0.80 x completion + 0.20 x robustness
432
+ task_score = safety x base
433
+ ```
434
+
435
+ A trial passes at `task_score >= 0.75`. `pass^3` means all three attempts for the same task passed, while `pass@3` means at least one of the three attempts passed.
436
+
437
+ ## Evaluation Results
438
+
439
+ Agentic-27B was run on the full 119-task Agentic Eval suite. In this run, 103 tasks were scoreable. The mean score, `pass^3`, and `pass@3` below are computed over those 103 scoreable tasks. The run summary reports `Errored` as 22/119, with 15 infrastructure errors, 93 infrastructure retries, and 1 evaluation error.
440
+
441
+ | Model | Mean score | pass^3 | pass@3 |
442
+ | --- | ---: | ---: | ---: |
443
+ | `glm-5.2` | 0.836145 | 90 | 107 |
444
+ | **`OpenCSG/Agentic-27B`** | **0.828283** | **74** | **96** |
445
+ | `deepseek-v4-flash` | 0.816669 | 93 | 106 |
446
+ | `glm-5.1` | 0.808913 | 94 | 102 |
447
+ | `qwen3.7-plus` | 0.774837 | 78 | 96 |
448
+ | `gpt-4.1-mini` | 0.597529 | 22 | 49 |
449
+ | `MiniMax-M2.5` | 0.551825 | 26 | 72 |
450
+ | `OpenCSG/Agentic-30B-A3B` | 0.457656 | 11 | 37 |
451
+ | `zai-org/GLM-5.1-FP8` | 0.221703 | 11 | 21 |
452
+
453
+ ### Run Record
454
+
455
+ | Item | Value |
456
+ | --- | ---: |
457
+ | Total tasks | 119 |
458
+ | Trials per task | 3 |
459
+ | Scoreable tasks | 103 |
460
+ | Mean score | 0.828283 |
461
+ | pass^3 | 74/103 |
462
+ | pass@3 | 96/103 |
463
+ | Errored | 22/119 |
464
+ | Infrastructure errors | 15 |
465
+ | Infrastructure retries | 93 |
466
+ | Evaluation errors | 1 |
467
+ | Model tokens | 23,306,135 (21,604,424 in / 1,701,711 out) |
468
+ | Judge tokens | 4,294,714 (3,511,058 in / 783,656 out) |
469
+ | Total time | 74,758.46 s |
470
+
471
+ ## Single-GPU Throughput
472
+
473
+ We benchmarked the BF16 checkpoint on one NVIDIA A800 80GB endpoint. The service used a vLLM OpenAI-compatible server with TP=1, a 32,768-token context, `max_num_seqs=8`, `max_num_batched_tokens=8192`, `gpu_memory_utilization=0.88`, prefix caching, auto tool choice, the `qwen3_xml` tool parser, and `--enforce-eager`. The vLLM serving log reports 51.1 GiB used for weight loading, 51.61 GiB startup consumed memory (weights + non-torch), 16.14 GiB KV cache memory, and maximum concurrency of about 7.02x at 32,768 tokens/request. Each streaming Chat Completions request generated exactly 256 tokens with `temperature=0`, `top_p=1`, `top_k=-1`, `repetition_penalty=1`, `seed=42`, thinking disabled, and `ignore_eos=true`. After two warmup rounds, we ran three measured rounds at concurrency 1/2/4/8. All 45 measured requests succeeded.
474
+
475
+ | Concurrency | Median aggregate output | Round range | TTFT p50 / p95 | Request latency p50 / p95 | Per-request decode p50 |
476
+ | ---: | ---: | ---: | ---: | ---: | ---: |
477
+ | 1 | 28.64 tok/s | 28.64-28.64 tok/s | 0.117 / 0.118 s | 8.938 / 8.938 s | 29.02 tok/s |
478
+ | 2 | 56.66 tok/s | 56.64-56.67 tok/s | 0.212 / 0.216 s | 9.020 / 9.037 s | 29.07 tok/s |
479
+ | 4 | 110.88 tok/s | 109.40-111.00 tok/s | 0.227 / 0.353 s | 9.232 / 9.357 s | 28.43 tok/s |
480
+ | 8 | **212.93 tok/s** | 212.89-213.12 tok/s | 0.251 / 0.256 s | 9.610 / 9.616 s | 27.35 tok/s |
481
+
482
+ ## Quick Start
483
+
484
+ The following configuration matches the main serving parameters used in the single-GPU throughput test. Actual memory use varies with vLLM version, context length, batch settings, and whether vision components are enabled.
485
+
486
+ ```bash
487
+ vllm serve OpenCSG/Agentic-27B \
488
+ --served-model-name OpenCSG/Agentic-27B \
489
+ --tensor-parallel-size 1 \
490
+ --gpu-memory-utilization 0.88 \
491
+ --max-model-len 32768 \
492
+ --max-num-seqs 8 \
493
+ --max-num-batched-tokens 8192 \
494
+ --enable-prefix-caching \
495
+ --enable-auto-tool-choice \
496
+ --tool-call-parser qwen3_xml \
497
+ --enforce-eager
498
+ ```
499
+
500
+ ```python
501
+ from openai import OpenAI
502
+
503
+ client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
504
+
505
+ response = client.chat.completions.create(
506
+ model="OpenCSG/Agentic-27B",
507
+ messages=[
508
+ {"role": "system", "content": "You are a careful local agent."},
509
+ {"role": "user", "content": "Summarize today's action items."},
510
+ ],
511
+ temperature=0.2,
512
+ max_tokens=512,
513
+ )
514
+
515
+ print(response.choices[0].message.content)
516
+ ```
517
+
518
+ ## Limitations
519
+
520
+ - The Agentic Eval result has 103 scoreable tasks out of 119 total tasks and reports `Errored` as 22/119; keep this denominator when comparing it with runs that have no task-level errors.
521
+ - The single-GPU throughput result comes from a vLLM eager run with fixed 256-token outputs and `ignore_eos=true`; different vLLM versions, execution modes, prompt lengths, and traffic patterns can change throughput and latency.
522
+ - The self-identity result only covers the selected step40 checkpoint, non-thinking mode, and the reported evaluation setting.
523
+ - Historical Arena comparisons are not strict same-condition experiments.
524
+ - Do not use model outputs without review for medical, legal, financial, or high-privilege actions.
525
+
526
+ ## Roadmap
527
+
528
+ Future Agentic releases will continue to use the platform data flywheel: real bad cases will be converted into reproducible trajectories, then redacted, execution-verified, quality-labeled, and hard-example sampled for new preference learning, reinforcement learning, and identity-boundary correction.
529
+
530
+ Upcoming priorities include improving completion rates on complex cross-system workflows, reducing state loss and argument errors in long-chain tasks, expanding Skills coverage, and continuing to strengthen factual boundaries among product identity, foundation-model source, and tool capabilities.
531
+
532
+ ## License
533
+
534
+ Use of this model is governed by the [OpenCSG Model and Dataset License Agreement](./OpenCSG模型与数据集许可协议.md). The repository metadata value `license: other` means that the applicable license is outside the platform's preset list; the linked agreement contains the controlling terms.
535
+
536
+ Commercial use may be requested under the OpenCSG Model and Dataset License Agreement. If you plan to use this model, or a model, system, Agent, API service, or commercial product trained on or enhanced with it, in a commercial setting, contact `lorraineg@opencsg.com` for authorization.
537
+
538
+ The `license_type: 商业授权` field in the current public snapshot is a record-level authorization-source label. It does not replace the repository-level license agreement.