Update README.md

e492163 verified 7 months ago

6.45 kB

---
license: apache-2.0
---

# MOP-RL-model

## Introduction

**MOP-RL-model** 是一款专为**多目标混合整数线性规划 (MO-MILP)** 任务打造的大型语言模型。该模型基于 Qwen2.5-7B 架构，并使用新颖的多目标规划强化学习框架 (MOP-RL) 进行了对齐训练。

在复杂的资源调度、智能制造与现代物流决策中，传统大模型往往难以平衡存在冲突的多目标，且容易在长序列推理中产生“逻辑幻觉”或“奖励作弊 (Reward Hacking)”。MOP-RL-model 突破了这一瓶颈，不仅掌握了基础的运筹学约束构建能力，更具备了在高维决策空间中捕捉帕累托前沿 (Pareto Front) 的深层权衡智慧。

本模型的核心技术亮点包括：

* 两阶段课程学习 (Two-stage Curriculum Learning)：模型经历了从“单目标基础训练 (密集奖励)”到“多目标扩展训练 (稀疏帕累托奖励)”的阶梯式对齐，有效抑制了强化学习中的策略震荡与基础能力崩塌。
* 帕累托感知奖励 (Pareto-Aware Reward Shaping)：摒弃了传统的标量逼近奖励，引入基于底层求解器 (如 Gurobi) 支配性测试的帕累托验证器，将复杂的数学证明转化为精确的绝对物理反馈。
* REINFORCE++ 算法：采用去除了价值网络 (Critic-free) 的改进型策略梯度算法，结合 Batch 内优势函数归一化与概率比裁剪，显著提升了面对上千 Token 的结构化思维链 (Structured CoT) 推理时的收敛稳定性。
* 结构化思维链输出 (Structured CoT)**：模型被强制要求遵循“问题分析——> 建模与标量化——>可执行代码生成”的严谨规范，确保了生成结果的物理可执行性与逻辑自治性。
## Requirements
运行 MOP-RL-model 的代码与普通 Qwen2.5 模型（详情参见：https://huggingface.co/Qwen/Qwen2.5-7B   ）一致，推荐使用最新版本的 `transformers`。
```bash
pip install transformers>=4.37.0
# 如果需要本地运行生成的优化代码，还需要安装求解器
pip install gurobipy
```
> **Warning**: The generated code often imports `gurobipy`. Ensure you have a valid Gurobi license to execute the generated solver scripts in your local environment.
## Quickstart
以下是一段使用 `transformers` 库加载并运行模型进行多目标建模的示例代码：
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "YourOrg/MOP-RL-model"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# 示例多目标调度问题描述
prompt = """
**[Task]**
Based on the [Problem Description] and [Target JSON Format] below, please write a complete Python script.
The script must use the `gurobipy` library to model and solve the multi-objective problem, and encapsulate the final solution results into JSON data, writing them to a file named `input.json`.
**[Input Information]**
1.  **Multi-objective Problem Description**:
    \"\"\"
    {problem_desc}
    \"\"\"
2.  **Target Output JSON Format Example**:
    \"\"\"
    {json_template}
    \"\"\"
**[Requirements]**
1.  **Modeling Logic**:
    - Clearly define decision variables and constraints.
    - **Multi-objective Handling**: Choose an appropriate multi-objective processing method based on the context (e.g., weighted sum, hierarchical sequence/lexicographic, or Pareto frontier). If weights are not specified, assume equal weights or provide adjustable parameters.
2.  **Code Standards**:
    - The code must be a complete, runnable Python script.
    - Include necessary comments explaining the mathematical model.
    - Must include checks for model solution status (e.g., checking for `GRB.OPTIMAL`).
3.  **Data Output**:
    - After solving, extract variable values.
    - **Format Matching**: Construct a Python dictionary that strictly matches the [Target Output JSON Format Example].
    - **File Writing**: Use `json.dump` to save the result as `input.json`.
4.  **Final Output**:
    - Please wrap your code in a Python markdown block, i.e., starts with ```python and ends with ```.
    - Provide only the Python code block."""
messages = [
    {"role": "system", "content": "You are an algorithm expert proficient in Operations Research and the Python Gurobi solver. You excel at translating complex business scenarios into mathematical models and writing robust code."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=2048
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

```
## RL code
RL训练代码放在如下链接里：
https://github.com/xuebozhang525-alt/MOP_RL
## Evaluation & Performance
MOP-RL-model 在具有极高难度的工业级多目标混合整数线性规划测试集（包含 1892 个复杂长尾案例）上进行了严苛的级联指标测评。
相较于直接在多目标数据上进行零样本推理或单纯进行 SFT 的基线模型，本模型实现了“降维打击”般的性能跃升。
| Metric / Evaluation Level | Description | Score |
| --- | --- | --- |
| 格式准确率  | 生成代码无格式错误 | 1.000 (100%) |
| 代码可执行率 | 无约束遗漏，成功提取变量并求得有效可行解，且求解器语法无报错的概率 | 88.01% |
| 条件帕累托率 | 在模型完全正确的前提下，解集通过支配性测试的概率 | 77.43% |
| 综合帕累托成功率  | parado解的总概率 | 68.15% |
与base的对比结果如下：
| 模型 | 格式准确率 | 代码可执行率 | 条件帕累托率 | 整体帕累托率 |
|------|-----------|------------|------------|------------|
| ChatGPT 5（闭源） | 0.985 | 0.615 | 0.732 | 45.0% |
| DeepSeek-R1(671B) | 0.968 | 0.821 | 0.582 | 47.8% |
| Qwen3-Max(1T) | 0.975 | 0.862 | 0.689 | 59.4% |
| MOP-RL(7B) (Ours) | 1.000 | 0.880 | 0.774 | 68.1% |
## Citation
如果您在研究中使用了本模型，请引用以下论文：
**📢 注：相关数据集和论文将会在近日发布。**
---

	---
	license: apache-2.0
	---


	# MOP-RL-model

	## Introduction

	MOP-RL-model 是一款专为多目标混合整数线性规划 (MO-MILP) 任务打造的大型语言模型。该模型基于 Qwen2.5-7B 架构，并使用新颖的多目标规划强化学习框架 (MOP-RL) 进行了对齐训练。

	在复杂的资源调度、智能制造与现代物流决策中，传统大模型往往难以平衡存在冲突的多目标，且容易在长序列推理中产生“逻辑幻觉”或“奖励作弊 (Reward Hacking)”。MOP-RL-model 突破了这一瓶颈，不仅掌握了基础的运筹学约束构建能力，更具备了在高维决策空间中捕捉帕累托前沿 (Pareto Front) 的深层权衡智慧。

	本模型的核心技术亮点包括：

	* 两阶段课程学习 (Two-stage Curriculum Learning)：模型经历了从“单目标基础训练 (密集奖励)”到“多目标扩展训练 (稀疏帕累托奖励)”的阶梯式对齐，有效抑制了强化学习中的策略震荡与基础能力崩塌。
	* 帕累托感知奖励 (Pareto-Aware Reward Shaping)：摒弃了传统的标量逼近奖励，引入基于底层求解器 (如 Gurobi) 支配性测试的帕累托验证器，将复杂的数学证明转化为精确的绝对物理反馈。
	* REINFORCE++ 算法：采用去除了价值网络 (Critic-free) 的改进型策略梯度算法，结合 Batch 内优势函数归一化与概率比裁剪，显著提升了面对上千 Token 的结构化思维链 (Structured CoT) 推理时的收敛稳定性。
	* 结构化思维链输出 (Structured CoT)**：模型被强制要求遵循“问题分析——> 建模与标量化——>可执行代码生成”的严谨规范，确保了生成结果的物理可执行性与逻辑自治性。

	## Requirements

	运行 MOP-RL-model 的代码与普通 Qwen2.5 模型（详情参见：https://huggingface.co/Qwen/Qwen2.5-7B ）一致，推荐使用最新版本的 `transformers`。

	```bash
	pip install transformers>=4.37.0
	# 如果需要本地运行生成的优化代码，还需要安装求解器
	pip install gurobipy

	```

	> Warning: The generated code often imports `gurobipy`. Ensure you have a valid Gurobi license to execute the generated solver scripts in your local environment.

	## Quickstart

	以下是一段使用 `transformers` 库加载并运行模型进行多目标建模的示例代码：

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model_name = "YourOrg/MOP-RL-model"

	model = AutoModelForCausalLM.from_pretrained(
	model_name,
	torch_dtype="auto",
	device_map="auto"
	)
	tokenizer = AutoTokenizer.from_pretrained(model_name)

	# 示例多目标调度问题描述
	prompt = """
	[Task]
	Based on the [Problem Description] and [Target JSON Format] below, please write a complete Python script.
	The script must use the `gurobipy` library to model and solve the multi-objective problem, and encapsulate the final solution results into JSON data, writing them to a file named `input.json`.

	[Input Information]
	1. Multi-objective Problem Description:
	\"\"\"
	{problem_desc}
	\"\"\"
	2. Target Output JSON Format Example:
	\"\"\"
	{json_template}
	\"\"\"

	[Requirements]
	1. Modeling Logic:
	- Clearly define decision variables and constraints.
	- Multi-objective Handling: Choose an appropriate multi-objective processing method based on the context (e.g., weighted sum, hierarchical sequence/lexicographic, or Pareto frontier). If weights are not specified, assume equal weights or provide adjustable parameters.
	2. Code Standards:
	- The code must be a complete, runnable Python script.
	- Include necessary comments explaining the mathematical model.
	- Must include checks for model solution status (e.g., checking for `GRB.OPTIMAL`).
	3. Data Output:
	- After solving, extract variable values.
	- Format Matching: Construct a Python dictionary that strictly matches the [Target Output JSON Format Example].
	- File Writing: Use `json.dump` to save the result as `input.json`.
	4. Final Output:
	- Please wrap your code in a Python markdown block, i.e., starts with ```python and ends with ```.
	- Provide only the Python code block."""

	messages = [
	{"role": "system", "content": "You are an algorithm expert proficient in Operations Research and the Python Gurobi solver. You excel at translating complex business scenarios into mathematical models and writing robust code."},
	{"role": "user", "content": prompt}
	]

	text = tokenizer.apply_chat_template(
	messages,
	tokenize=False,
	add_generation_prompt=True
	)
	model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

	generated_ids = model.generate(
	**model_inputs,
	max_new_tokens=2048
	)
	generated_ids = [
	output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
	]

	response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
	print(response)

	```

	## RL code

	RL训练代码放在如下链接里：
	https://github.com/xuebozhang525-alt/MOP_RL

	## Evaluation & Performance

	MOP-RL-model 在具有极高难度的工业级多目标混合整数线性规划测试集（包含 1892 个复杂长尾案例）上进行了严苛的级联指标测评。

	相较于直接在多目标数据上进行零样本推理或单纯进行 SFT 的基线模型，本模型实现了“降维打击”般的性能跃升。

	\| Metric / Evaluation Level \| Description \| Score \|
	\| --- \| --- \| --- \|
	\| 格式准确率 \| 生成代码无格式错误 \| 1.000 (100%) \|
	\| 代码可执行率 \| 无约束遗漏，成功提取变量并求得有效可行解，且求解器语法无报错的概率 \| 88.01% \|
	\| 条件帕累托率 \| 在模型完全正确的前提下，解集通过支配性测试的概率 \| 77.43% \|
	\| 综合帕累托成功率 \| parado解的总概率 \| 68.15% \|

	与base的对比结果如下：

	\| 模型 \| 格式准确率 \| 代码可执行率 \| 条件帕累托率 \| 整体帕累托率 \|
	\|------\|-----------\|------------\|------------\|------------\|
	\| ChatGPT 5（闭源） \| 0.985 \| 0.615 \| 0.732 \| 45.0% \|
	\| DeepSeek-R1(671B) \| 0.968 \| 0.821 \| 0.582 \| 47.8% \|
	\| Qwen3-Max(1T) \| 0.975 \| 0.862 \| 0.689 \| 59.4% \|
	\| MOP-RL(7B) (Ours) \| 1.000 \| 0.880 \| 0.774 \| 68.1% \|

	## Citation

	如果您在研究中使用了本模型，请引用以下论文：


	📢 注：相关数据集和论文将会在近日发布。

	---