StreetLearn

Overview

This repository contains an implementation of the StreetLearn C++ engine and Python environment for training navigation agents in real-world photographic street environments, as well as code for implementing the agents used in [1] "Learning to Navigate in Cities Without a Map" (NeurIPS 2018). This environment was also used in two follow-up papers: [2] "Cross-View Policy Learning for Street Navigation" (ICCV 2019) and [3] "Learning to follow directions in Street View" (AAAI 2020), as well as in technical report [4] "The StreetLearn Environment and Dataset". The StreetLearn environment relies on panorama images from Google Street View and provides an interface for moving a first-person view agent inside the Street View graph. This is not an officially supported Google product. Please cite papers [1] and [4] if you use the code from this repository in your work.

Our papers [1], [2] and [3] also provide a detailed description of how to train and implement navigation agents in the StreetLearn environment by using a TensorFlow implementation of "Importance Weighted Actor-Learner Architectures", published in Espeholt, Soyer, Munos et al. (2018) "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures"). The generic agent and trainer code have been published by Lasse Espeholt under an Apache license at: https://github.com/deepmind/scalable_agent.

Bibtex

@inproceedings{mirowski2018learning,
  title={Learning to Navigate in Cities Without a Map},
  author={Mirowski, Piotr and Grimes, Matthew Koichi and Malinowski, Mateusz and Hermann, Karl Moritz and Anderson, Keith and Teplyashin, Denis and Simonyan, Karen and Kavukcuoglu, Koray and Zisserman, Andrew and Hadsell, Raia},
  booktitle={Neural Information Processing Systems (NeurIPS)},
  year={2018}
}

@article{mirowski2019streetlearn,
  title={The StreetLearn Environment and Dataset},
  author={Mirowski, Piotr and Banki-Horvath, Andras and Anderson, Keith and Teplyashin, Denis and Hermann, Karl Moritz and Malinowski, Mateusz and Grimes, Matthew Koichi and Simonyan, Karen and Kavukcuoglu, Koray and Zisserman, Andrew and others},
  journal={arXiv preprint arXiv:1903.01292},
  year={2019}
}

Code structure

This environment code contains:

• streetlearn/engine Our C++ StreetLearn engine for loading, caching and serving Google Street View panoramas by projecting them from a equirectangular representation to first-person projected view at a given yaw, pitch and field of view, and for handling navigation (moving from one panorama to another) depending on the city street graph and the current orientation.
• streetlearn/proto The message protocol buffer used to store panoramas and street graph.
• streetlearn/python/environment A Python-based interface for calling the StreetLearn environment with custom action spaces. Within the Python StreetLearn interface, several games are defined in individual files whose names end with game.py. A second interface, called BatchedStreetLearn, can be used to instantiate multiple StreetLearn environments that share the same action specs, observation specs, and panorama cache, and return observations in batched format.
• streetlearn/python/ui A simple interactive human_agent and an oracle_agent and instruction_following_oracle_agent for courier and instruction-following tasks respectively; all agents are implemented in Python using pygame and instantiate the StreetLearn environment on the requested map, along with a simple user interface. The interactive human_agent enables a user to play various games. The oracle_agent and instruction_following_oracle_agent are similar to the human agent and automatically navigate towards the goal (courier game) or towards the goal via waypoints, following instructions (instruction-following game) and they report oracle performance on these tasks. A batched version of th oracle agent can be started using batched_oracle_agent.

Compilation from source

Bazel is the official build system for StreetLearn. The build has only been tested running on Ubuntu 18.04.

Install build prerequisites

sudo apt-get install autoconf automake libtool curl make g++ unzip virtualenv python-virtualenv cmake subversion pkg-config libpython-dev libcairo2-dev libboost-all-dev python-pip libssl-dev
pip install setuptools
pip install pyparsing

Install Protocol Buffers

For detailed information see: https://github.com/protocolbuffers/protobuf/blob/master/src/README.md

git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
./autogen.sh
./configure
make -j7
sudo make install
sudo ldconfig
cd python
python setup.py build
sudo python setup.py install
cd ../..

Install CLIF

git clone https://github.com/google/clif.git
cd clif
./INSTALL.sh
cd ..

Install OpenCV 2.4.13

wget https://github.com/opencv/opencv/archive/2.4.13.6.zip
unzip 2.4.13.6.zip
cd opencv-2.4.13.6
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j7
sudo make install
sudo ldconfig
cd ../..

Install Python dependencies

pip install six
pip install absl-py
pip install inflection
pip install wrapt
pip install numpy
pip install dm-sonnet
pip install tensorflow-gpu
pip install pygame

Install Bazel

This page describes how to install the Bazel build and test tool on your machine. We currently support Bazel versions up to 0.24.0, whose installation instructions are listed on this page under section Using the binary installer (copy-pasted below):

wget https://github.com/bazelbuild/bazel/releases/download/0.24.0/bazel-0.24.0-installer-linux-x86_64.sh
chmod +x bazel-0.24.0-installer-linux-x86_64.sh
./bazel-0.24.0-installer-linux-x86_64.sh  --user
export PATH="$PATH:$HOME/bin"

Building StreetLearn

Clone this repository and install Scalable Agent:

git clone https://github.com/deepmind/streetlearn.git
cd streetlearn
sh get_scalable_agent.sh

To build the StreetLearn engine only:

export CLIF_PATH=$HOME/opt
bazel build streetlearn:streetlearn_engine_py

To build the human agent and the oracle agent in the StreetLearn environment, with all the dependencies:

export CLIF_PATH=$HOME/opt
bazel build streetlearn/python/ui:all

Running the StreetLearn human agent

To run the human agent using one of the StreetLearn datasets downloaded and stored at dataset_path (note that dataset_path needs to be absolute, not relative):

bazel run streetlearn/python/ui:human_agent -- --dataset_path=<dataset_path>

For help with the options of the human_agent:

bazel run streetlearn/python/ui:human_agent -- --help

Similarly, to run the oracle agent on the courier game:

bazel run streetlearn/python/ui:oracle_agent -- --dataset_path=<dataset_path>

The human agent and the oracle agent show a view_image (on top) and a graph_image (on bottom).

Note: you need to add the following line to your .bashrc script to avoid specifying the CLIF path each time you open a new terminal:

export CLIF_PATH=$HOME/opt

Actions available to an agent:

• Rotate left or right in the panorama, by a specified angle (change the yaw of the agent). In the human_agent, press a or d.
• Rotate up or down in the panorama, by a specified angle (change the pitch of the agent). In the human_agent, press w or s.
• Move from current panorama A forward to another panorama B if the current bearing of the agent from A to B is within a tolerance angle of 30 degrees. In the human_agent, press space.
• Zoom in and out in the panorama. In the human_agent, press i or o.

Additional keys for the human_agent are escape and p (to print the current view as a bitmap image).

For training RL agents, action spaces are discretized using integers. For instance, in our paper, we used 5 actions: (move forward, turn left by 22.5 deg, turn left by 67.5 deg, turn right by 22.5 deg, turn right by 67.5 deg).

Navigation Bar

Along the bottom of the view_image is the navigation bar which displays a small circle in any direction in which travel is possible:

• When within the centre range, they will turn green meaning the user can move in this direction.
• When they are out of this range, they will turn red meaning this is inaccessible.
• When more than one dots are within the centre range, all except the most central will turn orange, meaning that there are multiple (forward) directions available.

Stop signs

The graph is constructed by breadth first search to the depth specified by the graph depth flags. At the maximum depth the graph will suddenly stop, generally in the middle of a street. Because we are trying to train agents to recognize streets as navigable, and in order not to confuse the agents, red stop signs are shown from two panoramas away from any terminal node in the graph.

Obtaining the StreetLearn dataset

You can request the StreetLearn dataset on the StreetLearn project website. The following datasets are currently distributed: