VLMaps

<b>Visual Language Maps for Robot Navigation</b>

Chenguang Huang, Oier Mees, Andy Zeng, Wolfram Burgard

We present VLMAPs (Visual Language Maps), a spatial map representation in which pretrained visuallanguage model features are fused into a 3D reconstruction of the physical world. Spatially anchoring visual language features enables natural language indexing in the map, which can be used to, e.g., localize landmarks or spatial references with respect to landmarks – enabling zero-shot spatial goal navigation without additional data collection or model finetuning.

Approach

Quick Start

Try VLMaps creation and landmark indexing in

Dependencies installation

To begin on your own machine, clone this repository locally

git clone https://github.com/vlmaps/vlmaps.git

Install requirements:

$ conda create -n vlmaps python=3.8 -y  # or use virtualenv
$ conda activate vlmaps
$ conda install jupyter -y
$ cd vlmaps
$ bash install.bash

Run Demo

git checkout demo

Start the jupyter notebook

$ jupyter notebook demo.ipynb

Advanced Usage

Use the master branch

git checkout master

Generate Dataset

To build VLMaps for simulated environments, we manually collected 10 sequences of RGB-D videos among 10 scenes in Habitat simulator with Matterport3D dataset. We provide script and pose meta data to generate the RGB-D videos. Please follow the next few steps to generate the dataset.

Download Matterpot3D dataset

Please check Dataset Download, sign the Terms of Use, and send to the responsible person to request the Matterport3D mesh for the use in Habitat simulator. The return email will attach a python script to download the data. Copy and paste the script to a file ~/download_mp.py. Run the following to download the data:

cd ~
# download the data at the current directory
python2 download_mp.py -o . --task habitat
# unzip the data
unzip v1/tasks/mp3d_habitat.zip
# the data_dir is mp3d_habitat/mp3d

Generate Dataset

• Configure the config/generate_dataset.yaml.

  • Change the value for defaults/data_paths in config/generate_dataset.yaml to default.
  • Change the habitat_scene_dir and vlmaps_data_dir in config/data_paths/default.yaml according to the following steps.
  • Change the habitat_scene_dir to the downloaded Matterport3D dataset ~/mp3d_habitat/mp3d.

    # the structure of the habitat_scene_dir looks like this
    habitat_scene_dir
      |-5LpN3gDmAk7
      |   |-5LpN3gDmAk7.glb
      |   |-5LpN3gDmAk7_semantic.ply
      |   |-...
      |-gTV8FGcVJC9
      |   |-gTV8FGcVJC9.glb
      |   |-gTV8FGcVJC9_semantic.ply
      |   |-...
      |-jh4fc5c5qoQ
      |   |-jh4fc5c5qoQ.glb
      |   |-jh4fc5c5qoQ_semantic.ply
      |   |-...
      ...
    

  • Change the vlmaps_data_dir to downloaded dataset

    # the structure of the vlmaps_data_dir looks like this
    vlmaps_data_dir
      |-5LpN3gDmAk7_1
      |   |-poses.txt
      |-gTV8FGcVJC9_1
      |   |-poses.txt
      |-jh4fc5c5qoQ_1
      |   |-poses.txt
      ...
    

  • Change data_cfg.resolution.w and data_cfg.resolution.h to adjust the resolution of the generated rgb, depth, and semantic images.
  • Change rgb, depth, and semantic to true to generate corresponding data, and to false to ignore corresponding data.
  • Change camera_height to change the height of camera relative to the robot base

  # go to <REPO_ROOT>/dataset of this repository
  cd dataset
  # the generated dataset will be in the same
  python generate_dataset.py
  

Collect your own data in Habitat-Sim

Run the following code to collect your own data (for example for scene gTV8FGcVJC9):

python dataset/collect_custom_dataset.py scene_names=["gTV8FGcVJC9"]

The generated data folder will be <scene_name>_<id> under vlmaps_data_dir in config/data_paths/default.yaml (If you already set the data_paths to default). The <scene_name> is gTV8FGcVJC9 for the above command, and the <id> depends on the existing data folders for this scene. If gTV8FGcVJC9_1 exists, then the new folder name will be gTV8FGcVJC9_2.

Create a VLMap with the Generated Dataset

• Change the value for defaults/data_paths in config/map_creation_cfg.yaml to default.

• Change the habitat_scene_dir and vlmaps_data_dir in config/data_paths/default.yaml according to the steps in the Generate Dataset section above.

• Run the following command to build the VLMap. The code builds a 3D map where each voxel contains the LSeg embedding.

  cd application
  python create_map.py
  

  

Config the Created VLMap

• Change the scene you want to generate VLMap for by changing scene_id (0-9) in config/map_creation_cfg.yaml. If you use your customized data, scene_id indicates the id of the sorted subfolder under vlmaps_data_dir path you set in config/paths/default.yaml.
• Customize the map by changing the parameters in config/params/default.yaml
  • Change the resolution of the map by changing cs (cell size in meter) and gs (grid size)
• Customize the camera pose and base pose by changing config/vlmaps.yaml. Change the pose_info section.
  • pose_type means the type of poses stored in poses.txt files. Set it to mobile_base when the poses are the poses for the base on the robot, and set it to camera_base when the poses are the poses for the camera.
  • rot_type means the format of data you save at each line of the poses.txt file. Set it to quat if you save the pose as px, py, pz, qx, qy, qz, qw. Set it to mat if you save the flattened (4, 4) transformation matrix at each line. By default, it is set to quat.
  • If you set pose_type as mobile_base, you should also modify the following parameters in pose_info:
    • camera_height means the camera height relative to the base. Change it if you set different camera height when you generate the dataset.
    • base2cam_rot means the row-wise flattened rotation matrix from robot base to the camera coordinate frame (z forward, x right, y down).
    • base_forward_axis, base_left_axis, base_up_axis: your robot base coordinate. They mean what is the coordinate of the forward unit vector [1, 0, 0] projected into your robot base frame, the coordinate of the left unit vector [0, 1, 0] projected into your robot base frame, the coordinate of the upward unit vector [0, 0, 1] projected into your robot base frame.
• Other settings in config/vlmaps.yaml
  • skip_frame means that only when frame_i % skip_frame == 0 do we use the frame
  • cam_calib_mat is the flattened camera intrinsics matrix
  • depth_sample_rate: we only back project randomly sampled h * w / depth_sample_rate pixels at each frame. You can change this to a higher value to increase the mapping speed at the cost of having a sparser point cloud at each frame.

Index a VLMap

• Change the value for defaults/data_paths in config/map_indexing_cfg.yaml to default.

• Change the habitat_scene_dir and vlmaps_data_dir in config/data_paths/default.yaml according to the steps in the Generate Dataset section above.

• Run the following command to index a VLMap you built

  cd application
  python index_map.py
  

  

Configure the Indexing

• Change the file config/map_indexing_cfg.yaml
  • decay_rate: set the heatmap decay rate. When it is smaller, the transition of the heat is clearer and covers larger area.
  • index_2d: set it to True to visualize 2D heatmap. Set it to False to visualize 3D heatmap.
  • init_categories: set it to True to provide a fix list of categories (MatterPort3D categories) to pick from indexing (for details you can check vlmaps/utils/matterport3d_categories.py). When you query with your text, the code will use GPT to find the nearest category in the fixed list. Set it to False to just use your input query for indexing. If you set it to True, you need to set an environment variable OPENAI_KEY to your OPENAI API tokens by running OPENAI_KEY=xxx in your terminal.

Test Navigation

Setup OpenAI

In order to test object goal navigation and spatial goal navigation tasks with our method, you need to setup an OpenAI API account with the following steps:

1. Sign up an OpenAI account, login your account, and bind your account with at least one payment method.
2. [Get