README.md

---
tags:
- reinforcement-learning
- pytorch
- custom-implementation
- ppo
- deep-reinforcement-learning
- gym
- lunar-lander
- arxiv:2604.13517
license: cc-by-4.0
library_name: pytorch
---

# Representation over Routing: Diagnosing Temporal Routing Pathologies in Multi-Timescale PPO

[![arXiv](https://img.shields.io/badge/arXiv-2604.13517-b31b1b.svg)](https://arxiv.org/abs/2604.13517)
[![GitHub](https://img.shields.io/badge/GitHub-Codebase-blue?logo=github)](https://github.com/ben-dlwlrma/Representation-Over-Routing)
[![Demo](https://img.shields.io/badge/Hugging%20Face-Space-yellow?logo=huggingface)](https://huggingface.co/spaces/ben-dlwlrma/Representation-Over-Routing-Demo)

This model repository hosts pretrained PyTorch actor weights for the diagnostic study **"Representation over Routing: Diagnosing Temporal Routing Pathologies in Multi-Timescale PPO"**.

The weights correspond to controlled PPO experiments on `LunarLander-v2`. They are provided to reproduce the qualitative behaviors discussed in the paper: a single-horizon baseline, differentiable temporal routing, error-based temporal routing, and Target Decoupling.

This model repository is a weight distribution package. Training scripts and selected generated figures live in the GitHub code repository; paper text and source files are distributed through arXiv.

## Related Links

- **Paper:** https://arxiv.org/abs/2604.13517
- **Interactive Demo Space:** https://huggingface.co/spaces/ben-dlwlrma/Representation-Over-Routing-Demo
- **GitHub Repository:** https://github.com/ben-dlwlrma/Representation-Over-Routing

## Model Weights Overview

The repository provides four standalone `.pth` actor weight files:

- **`1_baseline.pth` (Baseline PPO):** single-horizon PPO reference policy.
- **`2_surrogate_hacking_attention.pth` (Differentiable Routing):** policy from the actor-side attention routing diagnostic.
- **`3_temporal_paradox_variance.pth` (Error-Based Routing):** policy from the gradient-free error-based routing diagnostic.
- **`4_target_decoupling_final.pth` (Target Decoupling):** policy trained with structural separation between the actor objective and temporal routing. The actor uses the long-horizon advantage, while auxiliary critic heads remain as regularizers during training.

Target Decoupling is described in the paper as a structural isolation principle in the `LunarLander-v2` PPO setting. The reported evidence concerns removal of the actor-side routing pathway and improved observed worst-seed return in the tested run set, not broad benchmark superiority.

## Usage

For training scripts and selected diagnostic plots, see the [GitHub repository](https://github.com/ben-dlwlrma/Representation-Over-Routing). The manuscript itself is distributed through arXiv rather than duplicated as source files in the code or model repositories.

The published weights contain actor parameters and can be loaded into the same MLP actor architecture used by the training scripts:

```python
import torch
import torch.nn as nn
import numpy as np
import gymnasium as gym
from huggingface_hub import hf_hub_download

weight_path = hf_hub_download(
    repo_id="ben-dlwlrma/Representation-Over-Routing",
    filename="4_target_decoupling_final.pth",
)

def layer_init(layer, std=np.sqrt(2), bias_const=0.0):
    nn.init.orthogonal_(layer.weight, std)
    nn.init.constant_(layer.bias, bias_const)
    return layer

actor = nn.Sequential(
    layer_init(nn.Linear(8, 64)),
    nn.Tanh(),
    layer_init(nn.Linear(64, 64)),
    nn.Tanh(),
    layer_init(nn.Linear(64, 4), std=0.01),
)

actor.load_state_dict(torch.load(weight_path, weights_only=True))
actor.eval()

env = gym.make("LunarLander-v2")
state, _ = env.reset()
done = False

while not done:
    state_tensor = torch.FloatTensor(state).unsqueeze(0)
    with torch.no_grad():
        logits = actor(state_tensor)
        action = torch.argmax(logits, dim=1).item()

    state, reward, terminated, truncated, _ = env.step(action)
    done = terminated or truncated
```

The paper experiments were conducted on `LunarLander-v2`. The hosted demo may use `LunarLander-v3` for compatibility with current Gymnasium releases while preserving the same actor architecture and weight format.

## Citation

```bibtex
@misc{sunRepresentationRoutingDiagnosing2026,
  title = {Representation over {{Routing}}: {{Diagnosing Temporal Routing Pathologies}} in {{Multi-Timescale PPO}}},
  shorttitle = {Representation over {{Routing}}},
  author = {Sun, Jing},
  year = 2026,
  publisher = {arXiv},
  doi = {10.48550/ARXIV.2604.13517},
  urldate = {2026-04-16},
  copyright = {Creative Commons Attribution 4.0 International},
  keywords = {Artificial Intelligence (cs.AI),FOS: Computer and information sciences,Machine Learning (cs.LG)}
}
```