Sean13 commited on
Commit
d645199
·
verified ·
1 Parent(s): 6e450c7

Add README.md

Browse files
Files changed (1) hide show
  1. README.md +218 -0
README.md ADDED
@@ -0,0 +1,218 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - safe-multi-agent-reinforcement-learning
5
+ - constrained-optimization
6
+ - multi-agent-mujoco
7
+ - responsibility-decomposition
8
+ - mappo-lagrangian
9
+ - corl-2026
10
+ language: en
11
+ library_name: pytorch
12
+ pipeline_tag: reinforcement-learning
13
+ ---
14
+
15
+ # Learning Safety Burden Allocation by Equilibrium Response
16
+
17
+ This repository contains trained model checkpoints for the paper:
18
+
19
+ > **Learning Safety Burden Allocation by Equilibrium Response**
20
+ > Xiaoyang Cao
21
+ > CoRL 2026
22
+
23
+ ## Model Description
24
+
25
+ We propose a principled method for decomposing a joint safety constraint into per-agent responsibility shares in cooperative multi-agent reinforcement learning. Rather than using uniform splits or hand-tuned allocations, our approach **learns state-dependent responsibility vectors** via bilevel optimization, where the outer loop adjusts each agent's safety burden share (rho) to maximize team welfare, while the inner loop trains policies under MAPPO-Lagrangian with the assigned per-agent constraints.
26
+
27
+ The key insight is that different agents have different capacities to avoid costs at different states. By letting rho adapt to the state, agents who can cheaply reduce cost take on more safety burden, freeing capable agents to maximize reward.
28
+
29
+ ### Architecture
30
+
31
+ - **Inner loop**: MAPPO-Lagrangian (Multi-Agent PPO with Lagrangian constraint handling)
32
+ - Shared actor-critic networks (MLP, 2 layers, 256 hidden units)
33
+ - Per-agent Lagrange multiplier for constraint enforcement
34
+ - **Outer loop**: State-dependent rho network
35
+ - MLP mapping joint state to per-agent responsibility shares
36
+ - Softmax output to ensure shares sum to 1
37
+ - Updated via implicit differentiation through the inner equilibrium
38
+
39
+ ## Environments
40
+
41
+ | Environment | Agents | Description | Episodes |
42
+ |---|---|---|---|
43
+ | HalfCheetah 6x1 | 6 | Each agent controls 1 joint of a 6-joint cheetah | 1500 |
44
+ | Humanoid 9\|8 | 2 | Two agents control 9 and 8 joints respectively | 1000 |
45
+ | ManySegmentSwimmer 6x1 | 6 | Each agent controls 1 segment of a 6-segment swimmer | 1500 |
46
+ | Resource Harvest | 5 | Agents harvest resources with a shared sustainability constraint | 1500 |
47
+
48
+ ## Methods
49
+
50
+ - **Baseline (MAPPO-Lag)**: Standard MAPPO-Lagrangian with uniform safety budget split (rho_i = 1/n)
51
+ - **PAL (Per-Agent Lambda)**: Each agent maintains its own Lagrange multiplier against the joint constraint
52
+ - **Ours (Learned rho)**: State-dependent responsibility decomposition via bilevel optimization
53
+
54
+ ## Checkpoint Structure
55
+
56
+ ```
57
+ checkpoints/
58
+ halfcheetah_6x1/ # 6-agent HalfCheetah
59
+ baseline_s{0..4}.pt # 5 seeds, MAPPO-Lag uniform split
60
+ pal_s{0..4}.pt # 5 seeds, per-agent lambda
61
+ ours_s{0..4}.pt # 5 seeds, learned state-dep rho
62
+ humanoid_9_8/ # 2-agent Humanoid (9|8 partition)
63
+ baseline_s{0..4}.pt # 5 seeds
64
+ pal_s{0..3}.pt # 4 seeds
65
+ ours_s{0..4}.pt # 5 seeds
66
+ mss_6x1/ # 6-agent ManySegmentSwimmer
67
+ baseline_s{0..4}.pt # 5 seeds
68
+ pal_s{0..4}.pt # 5 seeds
69
+ ours_s{0..4}.pt # 5 seeds
70
+ harvest_5/ # 5-agent Resource Harvest
71
+ baseline_s{0..4}.pt # 5 seeds
72
+ pal_s{0..4}.pt # 5 seeds
73
+ ours_s{0..4}.pt # 5 seeds
74
+ ablation_hc6x1/ # Ablations on HalfCheetah 6x1
75
+ cg3_s{0..2}.pt # Conjugate gradient steps = 3
76
+ cg5_s{0..2}.pt # Conjugate gradient steps = 5
77
+ trho25_s{0..2}.pt # Rho update period T_rho = 25
78
+ trho100_s{0..2}.pt # Rho update period T_rho = 100
79
+ ```
80
+
81
+ Each `.pt` file is a dictionary containing:
82
+ - `actor_state_dict`: Policy network weights
83
+ - `critic_state_dict`: Value network weights
84
+ - `rho_net_state_dict` (ours only): Responsibility decomposition network
85
+ - `lambda_value`: Final Lagrange multiplier(s)
86
+ - `config`: Training hyperparameters
87
+ - `final_metrics`: Final evaluation metrics (reward, cost, constraint satisfaction)
88
+
89
+ ## How to Load and Evaluate
90
+
91
+ ```python
92
+ import torch
93
+ from huggingface_hub import hf_hub_download
94
+
95
+ # Download a checkpoint
96
+ ckpt_path = hf_hub_download(
97
+ repo_id="Sean13/responsibility-decomposition",
98
+ filename="checkpoints/halfcheetah_6x1/ours_s0.pt",
99
+ )
100
+
101
+ # Load
102
+ checkpoint = torch.load(ckpt_path, map_location="cpu")
103
+ print(checkpoint.keys())
104
+ # dict_keys(['actor_state_dict', 'critic_state_dict', 'rho_net_state_dict',
105
+ # 'lambda_value', 'config', 'final_metrics'])
106
+
107
+ # Inspect final metrics
108
+ print(f"Welfare: {checkpoint['final_metrics']['total_welfare']:.1f}")
109
+ print(f"Constraint satisfaction: {checkpoint['final_metrics']['constraint_satisfaction_pct']:.1f}%")
110
+ print(f"Final rho: {checkpoint['final_metrics']['final_rho']}")
111
+ ```
112
+
113
+ ### Run Evaluation
114
+
115
+ ```bash
116
+ # Clone this repo
117
+ git clone https://huggingface.co/Sean13/responsibility-decomposition
118
+ cd responsibility-decomposition
119
+
120
+ # Evaluate a checkpoint
121
+ python scripts/eval.py \
122
+ --checkpoint checkpoints/halfcheetah_6x1/ours_s0.pt \
123
+ --env mamujoco_HalfCheetah_6x1 \
124
+ --n_eval_episodes 100
125
+
126
+ # Render a video
127
+ python scripts/render.py \
128
+ --checkpoint checkpoints/halfcheetah_6x1/ours_s0.pt \
129
+ --env mamujoco_HalfCheetah_6x1 \
130
+ --output_dir videos/
131
+ ```
132
+
133
+ ## Training Details
134
+
135
+ ### Hyperparameters (HalfCheetah 6x1)
136
+
137
+ | Parameter | Value |
138
+ |---|---|
139
+ | Algorithm | MAPPO-Lagrangian |
140
+ | Learning rate (policy) | 5e-4 |
141
+ | Learning rate (critic) | 5e-4 |
142
+ | Learning rate (rho network) | 5e-3 |
143
+ | Learning rate (lambda) | 1e-2 |
144
+ | Hidden size | 256 |
145
+ | Layers | 2 |
146
+ | PPO clip | 0.2 |
147
+ | GAE lambda | 0.95 |
148
+ | Discount | 0.99 |
149
+ | Episode length | 1000 |
150
+ | Rollout threads | 10 |
151
+ | Training episodes | 1500 |
152
+ | Cost limit (per-step) | 1.0 |
153
+ | Rho update period (T_rho) | 50 |
154
+ | CG steps (implicit diff) | 10 |
155
+ | Activation | ReLU |
156
+
157
+ ### Compute
158
+
159
+ All experiments were run on MIT ORCD cluster using NVIDIA L40s GPUs (single GPU per run). Approximate wall-clock times:
160
+ - HalfCheetah 6x1: ~4 hours per seed
161
+ - Humanoid 9|8: ~3 hours per seed
162
+ - ManySegmentSwimmer 6x1: ~4 hours per seed
163
+ - Resource Harvest: ~2 hours per seed
164
+
165
+ Total compute: approximately 250 GPU-hours across all experiments and ablations.
166
+
167
+ ## How to Reproduce from Scratch
168
+
169
+ 1. Install the training codebase:
170
+ ```bash
171
+ git clone https://github.com/YOUR_USERNAME/corl_respon_vector
172
+ cd corl_respon_vector/macpo_base/MAPPO-Lagrangian
173
+ pip install -e .
174
+ ```
175
+
176
+ 2. Run training (example: HalfCheetah 6x1, ours, seed 0):
177
+ ```bash
178
+ python mappo_lagrangian/scripts/train/train_mujoco.py \
179
+ --env_name mujoco \
180
+ --scenario HalfCheetah-v2 \
181
+ --agent_conf 6x1 \
182
+ --algorithm_name mappo \
183
+ --seed 0 \
184
+ --n_rollout_threads 10 \
185
+ --num_env_steps 15000000 \
186
+ --episode_length 1000 \
187
+ --hidden_size 256 \
188
+ --layer_N 2 \
189
+ --lr 5e-4 \
190
+ --critic_lr 5e-4 \
191
+ --use_responsibility_decomposition \
192
+ --rho_mode state_dependent \
193
+ --lr_rho 5e-3 \
194
+ --rho_update_period 50 \
195
+ --cg_steps 10 \
196
+ --cost_limit 1.0
197
+ ```
198
+
199
+ See `configs/` for full per-environment configurations.
200
+
201
+ ## Citation
202
+
203
+ ```bibtex
204
+ @inproceedings{cao2026responsibility,
205
+ title={Learning Safety Burden Allocation by Equilibrium Response},
206
+ author={Cao, Xiaoyang},
207
+ booktitle={Conference on Robot Learning (CoRL)},
208
+ year={2026}
209
+ }
210
+ ```
211
+
212
+ ## License
213
+
214
+ This work is released under the [MIT License](https://opensource.org/licenses/MIT).
215
+
216
+ ## Acknowledgments
217
+
218
+ This research was supported by computational resources from the MIT Office of Research Computing and Data (ORCD).