File size: 6,454 Bytes

---
license: apache-2.0
---


# MOP-RL-model

## Introduction

**MOP-RL-model** 是一款专为**多目标混合整数线性规划 (MO-MILP)** 任务打造的大型语言模型。该模型基于 Qwen2.5-7B 架构，并使用新颖的多目标规划强化学习框架 (MOP-RL) 进行了对齐训练。

在复杂的资源调度、智能制造与现代物流决策中，传统大模型往往难以平衡存在冲突的多目标，且容易在长序列推理中产生“逻辑幻觉”或“奖励作弊 (Reward Hacking)”。MOP-RL-model 突破了这一瓶颈，不仅掌握了基础的运筹学约束构建能力，更具备了在高维决策空间中捕捉帕累托前沿 (Pareto Front) 的深层权衡智慧。

本模型的核心技术亮点包括：

* 两阶段课程学习 (Two-stage Curriculum Learning)：模型经历了从“单目标基础训练 (密集奖励)”到“多目标扩展训练 (稀疏帕累托奖励)”的阶梯式对齐，有效抑制了强化学习中的策略震荡与基础能力崩塌。
* 帕累托感知奖励 (Pareto-Aware Reward Shaping)：摒弃了传统的标量逼近奖励，引入基于底层求解器 (如 Gurobi) 支配性测试的帕累托验证器，将复杂的数学证明转化为精确的绝对物理反馈。
* REINFORCE++ 算法：采用去除了价值网络 (Critic-free) 的改进型策略梯度算法，结合 Batch 内优势函数归一化与概率比裁剪，显著提升了面对上千 Token 的结构化思维链 (Structured CoT) 推理时的收敛稳定性。
* 结构化思维链输出 (Structured CoT)**：模型被强制要求遵循“问题分析——> 建模与标量化——>可执行代码生成”的严谨规范，确保了生成结果的物理可执行性与逻辑自治性。

## Requirements

运行 MOP-RL-model 的代码与普通 Qwen2.5 模型（详情参见：https://huggingface.co/Qwen/Qwen2.5-7B   ）一致，推荐使用最新版本的 `transformers`。

```bash
pip install transformers>=4.37.0
# 如果需要本地运行生成的优化代码，还需要安装求解器
pip install gurobipy

```

> **Warning**: The generated code often imports `gurobipy`. Ensure you have a valid Gurobi license to execute the generated solver scripts in your local environment.

## Quickstart

以下是一段使用 `transformers` 库加载并运行模型进行多目标建模的示例代码：

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "YourOrg/MOP-RL-model"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 示例多目标调度问题描述
prompt = """
**[Task]**
Based on the [Problem Description] and [Target JSON Format] below, please write a complete Python script.
The script must use the `gurobipy` library to model and solve the multi-objective problem, and encapsulate the final solution results into JSON data, writing them to a file named `input.json`.

**[Input Information]**
1.  **Multi-objective Problem Description**:
    \"\"\"
    {problem_desc}
    \"\"\"
2.  **Target Output JSON Format Example**:
    \"\"\"
    {json_template}
    \"\"\"

**[Requirements]**
1.  **Modeling Logic**:
    - Clearly define decision variables and constraints.
    - **Multi-objective Handling**: Choose an appropriate multi-objective processing method based on the context (e.g., weighted sum, hierarchical sequence/lexicographic, or Pareto frontier). If weights are not specified, assume equal weights or provide adjustable parameters.
2.  **Code Standards**:
    - The code must be a complete, runnable Python script.
    - Include necessary comments explaining the mathematical model.
    - Must include checks for model solution status (e.g., checking for `GRB.OPTIMAL`).
3.  **Data Output**:
    - After solving, extract variable values.
    - **Format Matching**: Construct a Python dictionary that strictly matches the [Target Output JSON Format Example].
    - **File Writing**: Use `json.dump` to save the result as `input.json`.
4.  **Final Output**:
    - Please wrap your code in a Python markdown block, i.e., starts with ```python and ends with ```.
    - Provide only the Python code block."""

messages = [
    {"role": "system", "content": "You are an algorithm expert proficient in Operations Research and the Python Gurobi solver. You excel at translating complex business scenarios into mathematical models and writing robust code."},
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=2048
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

```

## RL code

RL训练代码放在如下链接里：
https://github.com/xuebozhang525-alt/MOP_RL

## Evaluation & Performance

MOP-RL-model 在具有极高难度的工业级多目标混合整数线性规划测试集（包含 1892 个复杂长尾案例）上进行了严苛的级联指标测评。

相较于直接在多目标数据上进行零样本推理或单纯进行 SFT 的基线模型，本模型实现了“降维打击”般的性能跃升。

| Metric / Evaluation Level | Description | Score |
| --- | --- | --- |
| 格式准确率  | 生成代码无格式错误 | 1.000 (100%) |
| 代码可执行率 | 无约束遗漏，成功提取变量并求得有效可行解，且求解器语法无报错的概率 | 88.01% |
| 条件帕累托率 | 在模型完全正确的前提下，解集通过支配性测试的概率 | 77.43% |
| 综合帕累托成功率  | parado解的总概率 | 68.15% |

与base的对比结果如下：

| 模型 | 格式准确率 | 代码可执行率 | 条件帕累托率 | 整体帕累托率 |
|------|-----------|------------|------------|------------|
| ChatGPT 5（闭源） | 0.985 | 0.615 | 0.732 | 45.0% |
| DeepSeek-R1(671B) | 0.968 | 0.821 | 0.582 | 47.8% |
| Qwen3-Max(1T) | 0.975 | 0.862 | 0.689 | 59.4% |
| MOP-RL(7B) (Ours) | 1.000 | 0.880 | 0.774 | 68.1% |

## Citation

如果您在研究中使用了本模型，请引用以下论文：


**📢 注：相关数据集和论文将会在近日发布。**

---