Reinforcement Learning Zoo

<br/> <a href="https://deepreinforcementlearningbook.org" target="\_blank"> <div align="center"> <img src="docs/img/rlzoo-logo.png" width="40%"/> </div> <!-- <div align="center"><caption>Slack Invitation Link</caption></div> --> </a> <br/>

RLzoo is a collection of the most practical reinforcement learning algorithms, frameworks and applications. It is implemented with Tensorflow 2.0 and API of neural network layers in TensorLayer2.0+, to provide a hands-on fast-developing approach for reinforcement learning practices and benchmarks. It supports basic toy-tests like OpenAI Gym and DeepMind Control Suite with very simple configurations. Moreover, RLzoo supports robot learning benchmark environment RLBench based on Vrep/Pyrep simulator. Other large-scale distributed training framework for more realistic scenarios with Unity 3D, Mujoco, Bullet Physics, etc, will be supported in the future. A Springer textbook is also provided, you can get the free PDF if your institute has Springer license.

Different from RLzoo for simple usage with high-level APIs, we also have a RL tutorial that aims to make the reinforcement learning tutorial simple, transparent and straight-forward with low-level APIs, as this would not only benefits new learners of reinforcement learning, but also provide convenience for senior researchers to testify their new ideas quickly.

<!-- <em>Gym: Atari</em> <em>Gym: Box2D </em> <em>Gym: Classic Control </em> <em>Gym: MuJoCo </em>-->

<img src="https://github.com/tensorlayer/RLzoo/blob/master/gif/atari.gif" height=250 width=205 > <img src="https://github.com/tensorlayer/RLzoo/blob/master/gif/box2d.gif" height=250 width=205 ><img src="https://github.com/tensorlayer/RLzoo/blob/master/gif/classic.gif" height=250 width=205 > <img src="https://github.com/tensorlayer/RLzoo/blob/master/gif/mujoco.gif" height=250 width=205 >

<!-- <em>Gym: Robotics</em> <em>DeepMind Control Suite </em> <em>Gym: RLBench </em> -->

<img src="https://github.com/tensorlayer/RLzoo/blob/master/gif/robotics.gif" height=250 width=205 > <img src="https://github.com/tensorlayer/RLzoo/blob/master/gif/dmcontrol.gif" height=250 width=205 > <img src="https://github.com/tensorlayer/RLzoo/blob/master/gif/rlbench.gif" height=250 width=205 > <img src="https://github.com/tensorlayer/tensorlayer/blob/master/img/tl_transparent_logo.png" height=180 width=200 >

Please check our Online Documentation for detailed usage and the Arxiv paper for the high-level description of design choices plus comparisons with other RL libraries. We suggest users to report bugs using Github issues. Users can also discuss how to use RLzoo in the following slack channel.

<br/> <a href="https://join.slack.com/t/tensorlayer/shared_invite/enQtODk1NTQ5NTY1OTM5LTQyMGZhN2UzZDBhM2I3YjYzZDBkNGExYzcyZDNmOGQzNmYzNjc3ZjE3MzhiMjlkMmNiMmM3Nzc4ZDY2YmNkMTY" target="\_blank"> <div align="center"> <img src="https://github.com/tensorlayer/tensorlayer/raw/master/img/join_slack.png" width="40%"/> </div> </a> <br/>

[News] RLzoo's paper is accepted at ACM Multimedia 2021 Open Source Software Competition! See a simple presentation slide describing the key characteristics of RLzoo.

Table of contents:

• Status
• Installation
• Prerequisites
• Usage
• Contents
  • Algorithms
  • Environments
  • Configurations
• Properties
• Troubleshooting
• Credits
• Citing

Status: Release

<details><summary><b>Current status</b> <i>[click to expand]</i></summary> <div> We are currently open to any suggestions or pull requests from the community to make RLzoo a better repository. Given the scope of this project, we expect there could be some issues over the coming months after initial release. We will keep improving the potential problems and commit when significant changes are made in the future. Current default hyperparameters for each algorithm and each environment may not be optimal, so you can play around with those hyperparameters to achieve best performances. We will release a version with optimal hyperparameters and benchmark results for all algorithms in the future. </div> </details> <details><summary><b>Version History</b> <i>[click to expand]</i></summary> <div>

• 1.0.4 (Current version)

  Changes:

  • Add distributed training for DPPO algorithm, using Kungfu

• 1.0.3

  Changes:

  • Fix bugs in SAC algorithm

• 1.0.1

  Changes:

  • Add interactive training configuration;
  • Better support RLBench environment, with multi-head network architectures to support dictionary as observation type;
  • Make the code cleaner.

• 0.0.1

</div> </details>

Installation

Ensure that you have Python >=3.5 (Python 3.6 is needed if using DeepMind Control Suite).

Direct installation:

pip3 install rlzoo --upgrade

Install RLzoo from Git:

git clone https://github.com/tensorlayer/RLzoo.git
cd RLzoo
pip3 install .

Prerequisites

pip3 install -r requirements.txt

<details><summary><b>List of prerequisites.</b> <i>[click to expand]</i></summary> <div>

• tensorflow >= 2.0.0 or tensorflow-gpu >= 2.0.0a0
• tensorlayer >= 2.0.1
• tensorflow-probability
• tf-nightly-2.0-preview
• Mujoco 2.0, dm_control, dm2gym (if using DeepMind Control Suite environments)
• Vrep, PyRep, RLBench (if using RLBench environments, follows here, here and here)

</div> </details>

Usage

For detailed usage, please check our online documentation.

Quick Start

Choose whatever environments with whatever RL algorithms supported in RLzoo, and enjoy the game by running following example in the root file of installed package:

# in the root folder of RLzoo package
cd rlzoo
python run_rlzoo.py

What's in run_rlzoo.py?

from rlzoo.common.env_wrappers import build_env
from rlzoo.common.utils import call_default_params
from rlzoo.algorithms import TD3  # import the algorithm to use
# choose an algorithm
AlgName = 'TD3'
# chose an environment
EnvName = 'Pendulum-v0'  
# select a corresponding environment type
EnvType = 'classic_control'
# build an environment with wrappers
env = build_env(EnvName, EnvType)  
# call default parameters for the algorithm and learning process
alg_params, learn_params = call_default_params(env, EnvType, AlgName)  
# instantiate the algorithm
alg = eval(AlgName+'(**alg_params)')
# start the training
alg.learn(env=env, mode='train', render=False, **learn_params)  
# test after training 
alg.learn(env=env, mode='test', render=True, **learn_params)  

The main script run_rlzoo.py follows (almost) the same structure for all algorithms on all environments, see the full list of examples.

General Descriptions: RLzoo provides at least two types of interfaces for running the learning algorithms, with (1) implicit configurations or (2) explicit configurations. Both of them start learning program through running a python script, instead of running a long command line with all configurations shortened to be arguments of it (e.g. in Openai Baseline). Our approaches are found to be more interpretable, flexible and convenient to apply in practice. According to the level of explicitness of learning configurations, we provided two different ways of setting learning configurations in python scripts: the first one with implicit configurations uses a default.py script to record all configurations for each algorithm, while the second one with explicit configurations exposes all configurations to the running scripts. Both of them can run any RL algorithms on any environments supported in our repository with a simple command line.

<details><summary><b>1. Implicit Configurations</b> <i>[click to expand]</i></summary> <div>

RLzoo with implicit configurations means the configurations for learning are not explicitly contained in the main script for running (i.e. run_rlzoo.py), but in the default.py file in each algorithm folder (for example, rlzoo/algorithms/sac/default.py is the default parameters configuration for SAC algorithm). All configurations include (1) parameter values for the algorithm and learning process, (2) the network structures, (3) the optimizers, etc, are divided into configurations for the algorithm (stored in alg_params) and configurations for the learning process (stored in learn_params). Whenever you want to change the configurations for the algorithm or learning process, you can either go to the folder of each algorithm and mo