This repository has been deprecated in favor of the Retro (https://github.com/openai/retro) library. See our Retro Contest (https://blog.openai.com/retro-contest) blog post for detalis.

universe

---

Universe <https://openai.com/blog/universe/>_ is a software platform for measuring and training an AI's general intelligence across the world's supply of games, websites and other applications. This is the universe open-source library, which provides a simple Gym <https://github.com/openai/gym>__ interface to each Universe environment.

Universe allows anyone to train and evaluate AI agents on an extremely wide range of real-time, complex environments.

Universe makes it possible for any existing program to become an OpenAI Gym environment, without needing special access to the program's internals, source code, or APIs. It does this by packaging the program into a Docker container, and presenting the AI with the same interface a human uses: sending keyboard and mouse events, and receiving screen pixels. Our initial release contains over 1,000 environments in which an AI agent can take actions and gather observations.

Additionally, some environments include a reward signal sent to the agent, to guide reinforcement learning. We've included a few hundred environments with reward signals. These environments also include automated start menu clickthroughs, allowing your agent to skip to the interesting part of the environment.

We'd like the community's help <https://openai.com/blog/universe/#help>_ to grow the number of available environments, including integrating increasingly large and complex games.

The following classes of tasks are packaged inside of publicly-available Docker containers, and can be run today with no work on your part:

• Atari and CartPole environments over VNC: gym-core.Pong-v3, gym-core.CartPole-v0, etc.
• Flashgames over VNC: flashgames.DuskDrive-v0, etc.
• Browser tasks ("World of Bits") over VNC: wob.mini.TicTacToe-v0, etc.

We've scoped out integrations for many other games, including completing a high-quality GTA V integration (thanks to Craig Quiter <http://deepdrive.io/>_ and NVIDIA), but these aren't included in today's release.

.. contents:: Contents of this document :depth: 2

Getting started

Installation

Supported systems

We currently support Linux and OSX running Python 2.7 or 3.5.

We recommend setting up a `conda environment <http://conda.pydata.org/docs/using/envs.html>`__
before getting started, to keep all your Universe-related packages in the same place.

Install Universe
~~~~~~~~~~~~~~~~
To get started, first install ``universe``:

.. code:: shell

    git clone https://github.com/openai/universe.git
    cd universe
    pip install -e .

If this errors out, you may be missing some required packages. Here's
the list of required packages we know about so far (please let us know
if you had to install any others).

On Ubuntu 16.04:

.. code:: shell

    pip install numpy
    sudo apt-get install golang libjpeg-turbo8-dev make

On Ubuntu 14.04:

.. code:: shell

    sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable  # for newer golang
    sudo apt-get update
    sudo apt-get install golang libjpeg-turbo8-dev make

On OSX:

You might need to install Command Line Tools by running:

.. code:: shell

    xcode-select --install

Or ``numpy``, ``libjpeg-turbo`` and ``incremental`` packages:

.. code:: shell

    pip install numpy incremental
    brew install golang libjpeg-turbo

Install Docker
~~~~~~~~~~~~~~

The majority of the environments in Universe run inside Docker
containers, so you will need to `install Docker
<https://docs.docker.com/engine/installation/>`__ (on OSX, we
recommend `Docker for Mac
<https://docs.docker.com/docker-for-mac/>`__). You should be able to
run ``docker ps`` and get something like this:

.. code:: shell

     $ docker ps
     CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Alternate configuration - running the agent in docker

The above instructions result in an agent that runs as a regular python process in your OS, and launches docker containers as needed for the remotes. Alternatively, you can build a docker image for the agent and run it as a container as well. You can do this in any operating system that has a recent version of docker installed, and the git client.

To get started, clone the universe repo:

.. code:: shell

git clone https://github.com/openai/universe.git
cd universe

Build a docker image, tag it as 'universe':

.. code:: shell

docker build -t universe .

This may take a while the first time, as the docker image layers are pulled from docker hub.

Once the image is built, you can do a quick run of the test cases to make sure everything is working:

.. code:: shell

docker run --privileged --rm -e DOCKER_NET_HOST=172.17.0.1 -v /var/run/docker.sock:/var/run/docker.sock universe pytest

Here's a breakdown of that command:

• docker run - launch a docker container
• --rm - delete the container once the launched process finishes
• -e DOCKER_NET_HOST=172.17.0.1 - tells the universe remote (when launched) to make its VNC connection back to this docker-allocated IP
• -v /var/run/docker.sock:/var/run/docker.sock - makes the docker unix socket from the host available to the container. This is a common technique used to allow containers to launch other containers alongside itself.
• universe - use the imaged named 'universe' built above
• pytest - run 'pytest' in the container, which runs all the tests

At this point, you'll see a bunch of tests run and hopefully all pass.

To do some actual development work, you probably want to do another volume map from the universe repo on your host into the container, then shell in interactively:

.. code:: shell

docker run --privileged --rm -it -e DOCKER_NET_HOST=172.17.0.1 -v /var/run/docker.sock:/var/run/docker.sock -v (full path to cloned repo above):/usr/local/universe universe python

As you edit the files in your cloned git repo, they will be changed in your docker container and you'll be able to run them in python.

Note if you are using docker for Windows, you'll need to enable the relevant shared drive for this to work.

Notes on installation

* When installing ``universe``, you may see ``warning`` messages.  These lines occur when installing numpy and are normal.
* You'll need a ``go version`` of at least 1.5. Ubuntu 14.04 has an older Go, so you'll need to `upgrade <https://golang.org/doc/install>`_ your Go installation.
* We run Python 3.5 internally, so the Python 3.5 variants will be much more thoroughly performance tested. Please let us know if you see any issues on 2.7.
* While we don't officially support Windows, we expect our code to be very close to working there. We'd be happy to take pull requests that take our Windows compatibility to 100%. In the meantime, the easiest way for Windows users to run universe is to use the alternate configuration described above.

System overview
---------------

A Universe **environment** is similar to any other Gym environment:
the agent submits actions and receives observations using the ``step()``
method.

Internally, a Universe environment consists of two pieces: a **client** and a **remote**:

* The **client** is a `VNCEnv
  <https://github.com/openai/universe/blob/master/universe/envs/vnc_env.py>`_
  instance which lives in the same process as the agent. It performs
  functions like receiving the agent's actions, proxying them to the
  **remote**, queuing up rewards for the agent, and maintaining a
  local view of the current episode state.
* The **remote** is the running environment dynamics, usually a
  program running inside of a Docker container. It can run anywhere --
  locally, on a remote server, or in the cloud. (We have a separate
  page describing how to manage `remotes <doc/remotes.rst>`__.)
* The client and the remote communicate with one another using the
  `VNC <https://en.wikipedia.org/wiki/Virtual_Network_Computing>`__
  remote desktop system, as well as over an auxiliary WebSocket
  channel for reward, diagnostic, and control messages. (For more
  information on client-remote communication, see the separate page on
  the `Universe internal communication protocols
  <doc/protocols.rst>`__.)

The code in this repository corresponds to the **client** side of the
Universe environments. Additionally, you can freely access the Docker
images for the **remotes**. We'll release the source repositories for
the remotes in the future, along with tools to enable users to
integrate new environments. Please sign up for our `beta
<https://docs.google.com/forms/d/e/1FAIpQLScAiW-kIS0mz6hdzzFZJJFlXlicDvQs1TX9XMEkipNwjV5VlA/viewform>`_
if you'd like early access.

Run your first agent
--------------------

Now that you've installed the ``universe`` library, you should make
sure it actually works. You can paste the example below into your
``python`` REPL. (You may need to press enter an extra time to make
sure the ``while`` loop is executing.)

.. code:: python

  import gym
  import universe  # register the universe environments

  env = gym.make('flashgames.DuskDrive-v0')
  env.configure(remotes=1)  # automatically creates a local docker container
  observation_n = env.reset()

  while True:
    action_n = [[('KeyEvent', 'ArrowUp', True)] for ob in observation_n]  # your agent here
    observation_n, reward_n, done_n, info = env.step(action_n)
    env.render()

The example will instantiate a client in your Python process,
automatically pull the ``quay.io/openai/universe.flashgames`` image,
and will start that image as the remote. (In our `remotes
<doc/remotes.rst>`__ documentation page, we explain other ways you can run
remotes.)

It will t