| --- |
| license: cc-by-4.0 |
| tags: |
| - reinforcement-learning |
| - multi-objective |
| - production-scheduling |
| - attention |
| - ppo |
| - morl |
| language: |
| - en |
| pipeline_tag: reinforcement-learning |
| --- |
| |
| # SmartFlexPack MORL: Sequential Attention Policy for Packaging Scheduling |
|
|
| Trained model weights for the paper: |
|
|
| > **Multi-Objective Reinforcement Learning for Adaptive Packaging Scheduling via Sequential Attention Policy** |
| > L. Venturoso, G. Egger, F. Visentin, M. Todescato |
| > Fraunhofer Italia Research EFRE1054 Smartflexpack |
| > *Manuscript in preparation* |
|
|
| ## Model description |
|
|
| `SeqAttentionActorCritic` (12,355 parameters) decides, for each order, between |
| inline printing and labelling. At each planning event a one-layer multi-head |
| self-attention encoder (4 heads, d=32) encodes the pending orders; for each |
| decision a 54→64→64 MLP combines that embedding with ten live decision features |
| and twelve global features, including the preference vector |
| `(tardiness, cost, CO2)`, and outputs two action logits and a value. |
|
|
| A single trained policy covers the whole trade-off: the operator sets the |
| preference at inference time, with no retraining. |
|
|
| ## Results |
|
|
| In the reference plant configuration (3 label lines, inline 0.45 s/pack) the |
| policy exceeds the hypervolume of the full family of 32 realisable reference |
| controllers by **+0.058** (95% CI [+0.025, +0.095], about 7%), replicated on an |
| independent seed set (+0.050). The advantage grows with a slower inline line |
| (12–15% at 0.60 s/pack); with a single label line, and with the fastest inline |
| line, the policy ties the reference family or falls slightly below it. |
|
|
| ## Repository structure |
|
|
| ``` |
| ppo/seed_{1..5}/policy_200.pt # final PPO checkpoints (5 independent seeds) |
| ppo/seed_{1..5}/metrics.csv # training metrics |
| bc/bc_seq_morl.pt # behavioural-cloning warm start |
| configs/paper.json # training configuration |
| normalisation/ # observation and reward normalisers |
| inference/seq_morl/ # minimal code to load the policy |
| manifest.json # source revision and SHA-256 of every file |
| ``` |
|
|
| ## Inference |
|
|
| ```python |
| import sys, torch |
| from pathlib import Path |
| from huggingface_hub import snapshot_download |
| |
| release = Path(snapshot_download("fhi/smartflexpack-morl")) |
| sys.path.insert(0, str(release / "inference")) |
| from seq_morl.policies.attention import SeqAttentionActorCritic |
| |
| model = SeqAttentionActorCritic() |
| model.load_state_dict(torch.load(release / "ppo/seed_1/policy_200.pt", |
| map_location="cpu", weights_only=True)) |
| model.eval() |
| |
| with torch.no_grad(): |
| ctx = model.encode_sequence(order_feats) # [N, 32], pending orders |
| logits, value = model.forward_single(ctx[i], extra) # extra: 10 decision + 12 global |
| action = logits.argmax(dim=-1).item() # 0 = inline, 1 = label |
| ``` |
|
|
| Simulation and evaluation: `experiments/evaluate_policy.py` in the project |
| repository; it also needs the separate DataGenerator. |
|
|
| ## Training |
|
|
| Behavioural-cloning warm start, then 5 independent seeds of 200 PPO updates, |
| 10 parallel episodes of 600 orders, lr=3e-4 (Adam). About 3.8 h CPU per seed. |
|
|
| Training code: `experiments/train_bc_warm_start.py`, `experiments/train_policy.py` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @unpublished{venturoso2026morl, |
| title = {Multi-Objective Reinforcement Learning for Adaptive Packaging |
| Scheduling via Sequential Attention Policy}, |
| author = {Venturoso, Leonardo and Egger, Georg and |
| Visentin, Francesco and Todescato, Marco}, |
| year = {2026}, |
| note = {Manuscript in preparation} |
| } |
| ``` |
|
|
| ## License |
|
|
| [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| Fraunhofer Italia Research, EFRE1054 Smartflexpack project |
| (funded under FESR 2021–27, Priority 1 "Smart" R&I programme). |
|
|
