| --- |
| title: Upside-Down Reinforcement Learning |
| emoji: 🤖 |
| colorFrom: green |
| colorTo: gray |
| sdk: streamlit |
| python_version: "3.10" |
| sdk_version: 1.39.0 |
| app_file: app.py |
| pinned: true |
| short_description: Upside-Down Reinforcement Learning (UDRL) |
| tags: |
| - reinforcement learning |
| - RL |
| - upside-down reinforcement learning |
| - interpretability |
| - explainable AI |
| --- |
| |
| # Upside-Down RL |
|
|
| <img alt="Version" src="./website_photo.jpg" /> |
|
|
|
|
| This project implements an Upside-Down Reinforcement Learning (UDRL) agent. |
|
|
| This is the codebase of the paper: [arXiv](https://arxiv.org/abs/2411.11457) |
|
|
| The website associated with it is: [demo](https://vimmoos-udrl.hf.space/) |
|
|
|
|
| ### Installation |
|
|
| 1. Make sure you have Python 3.10 installed. You can check your version with `python --version`. |
| **NOTE** Use a virtual env to avoid dependency clash |
| 2. Install the project dependencies using Poetry: |
| ```bash |
| poetry install |
| ``` |
| If you do not have poetry use pip to install the requirements like so: |
| ```bash |
| pip install -r requirements.txt |
| ``` |
|
|
|
|
| ### Running the Experiment |
|
|
| You can run the experiment with various configuration options using the command line: |
|
|
| ```bash |
| poetry run python -m udrl [options] |
| ``` |
| **Note** If you are already inside a virtual env `python -m udrl [options]` is enough |
| **Note** All defaults are for the CartPole-v0 |
| Available options include: |
|
|
| * `--env_name`: Name of the Gym environment (default: `CartPole-v0`) |
| * `--estimator_name`: "neural" for NN or a fully qualified name of the scikit-learn estimator class (default: `ensemble.RandomForestClassifier`) |
| * `--seed`: Random seed (default: `42`) |
| * `--max_episode`: Maximum training episodes (default: `500`) |
| * `--collect_episode`: Episodes to collect between training (default: `15`) |
| * `--batch_size`: Batch size for training (default: `0`, uses entire replay buffer) |
| * Other options related to warm-up, memory size, exploration, testing, saving, etc. |
|
|
|
|
| ### Result Data |
|
|
| * Experiment configuration and final test results are saved in a JSON file (`conf.json`) within a directory structure based on the environment, seed, and non-default configuration values (e.g., `data/[env-name]/[experiment_name]/[seed]/conf.json`). |
| * If `save_policy` is True, the trained policy is saved in the same directory (`policy`). |
| * If `save_learning_infos` is True, learning infos and rewards during training are saved as a NumPy file (e.g.`test_rewards.npy`) and a json file (e.h.`learning_infos.json`) in the same directory. |
|
|
| ### Process Data |
| * A base post processing is available to convert the results data in csvs run it as `python -m udrl.data_proc` |
|
|
| ### Project Structure |
|
|
| * `data`: Stores experiment results and other data. |
| * `old_code`: Contains previous code versions (not used in the current setup). |
| * `poetry.lock`, `pyproject.toml`: Manage project dependencies and configuration. |
| * `README.md`: This file. |
| * `udrl`: Contains the main Python modules for the UDRL agent. |
|
|
| Please refer to the code and comments for further details on the implementation. |
|
|
|
|
|
|
| ## Troubleshooting |
|
|
| If you encounter any errors during installation or execution, or if you have any questions about the project, feel free to reach out to me at [massimiliano@falzari.dev](mailto:massimiliano@falzari.dev) or open an issue. I'll be happy to assist you! |
|
|
