LTU-AI/hdppo-Pendulum-v1

HD-PPO Agent playing Pendulum-v1

This is a trained HD-PPO (Hyperdimensional Proximal Policy Optimization) agent playing Pendulum-v1 using multi-band Fractional Power Encoding (FPE) with a continuous Gaussian actor.

Published by LTU-AI.

The policy was produced by a prune-and-fine-tune pipeline:

1. Train a multi-band teacher at D=512 with beta bands [1.25, 5.0].
2. Prune per-band through D=256 → 128 → 64 → 32 → 16.
3. Fine-tune each pruned checkpoint with PPO (log_std / entropy reset).

The published checkpoint is the D=16 fine-tuned model.

Usage

Install dependencies:

pip install -r requirements.txt

Evaluate the local checkpoint:

python enjoy.py --weights hdppo-Pendulum-v1/weights.npz --episodes 10

Render episodes in a window:

python enjoy.py --weights hdppo-Pendulum-v1/weights.npz --render --episodes 3

Record a replay video:

python record_video.py --weights hdppo-Pendulum-v1/weights.npz --output replay.mp4

Load from Hugging Face Hub:

python enjoy.py --weights LTU-AI/hdppo-Pendulum-v1 --episodes 10

Training pipeline

Reproduce the full teacher → prune → fine-tune workflow:

python pendulum_prune_finetune.py

Hyperparameters

{
    "env": "Pendulum-v1",
    "algo": "HD-PPO (multi-band FPE, continuous)",
    "teacher_D": 512,
    "pruned_D": 16,
    "beta_bands": [1.25, 5.0],
    "teacher_timesteps": 3000000,
    "finetune_timesteps": 2000000,
    "rollout_steps": 1024,
    "actor_lr": 0.003,
    "critic_lr": 0.005,
    "n_epochs": 8,
    "entropy_coef": 0.1,
    "clip_eps": 0.20,
    "gamma": 0.95,
    "lam": 0.95,
    "solve_thresh": -200,
}

Environment Arguments

{"render_mode": "rgb_array"}

Model files

File	Description
hdppo-Pendulum-v1/weights.npz	Actor, critic, and FPE encoder weights (D=16)
replay.mp4	Sample rollout video
results.json	Evaluation summary
config.yml	Training hyperparameters

Citation

If you use this model, please cite the HD-PPO work and acknowledge the prune-and-fine-tune recipe used for Pendulum-v1.