DSAC++ Code Documentation

• General Setup
• Programs
• Data Structure

Introduction

This document explains the high level concepts and setup of our improved DSAC code for camera localization. See it in action here, and also see the project page. You find an explanation of the method and theory in the following paper. Please cite this paper if you use this code in your own work:

E. Brachmann, C. Rother,
”Learning Less is More - 6D Camera Localization via 3D Surface Regression”,
CVPR 2018

We publish the code under the BSD License 2.0. The predecessor of this pipeline can be found here.

Compiling the code creates four programs:

• train_obj: Trains a scene coordinate regression CNN by optimizing the 3D distance between target scene coordinates and the CNN predictions (see Sec. 2.4. in our paper, "Scene Coordinate Initalization"). Target scene coordinates are either calculated from depth images (measured, rendered or estimated) or using our scene coordinate heuristic. The method requires RGB images, ground truth camera poses, and (optionally) depth images.
• train_repro: Trains a scene coordinate regression CNN by optimizing the reprojection error of the CNN predictions (see Sec. 2.4. in our paper, "Optimization of Reprojection Error"). The method requires RGB images and ground truth camera poses, and a CNN model initialized e.g. by train_obj.
• train_ransac: Trains the whole camera localization pipeline end-to-end using DSAC (see Sec. 2.4. in our paper, "End-to-End Optimization."). The method requires RGB images and ground truth camera poses, and a CNN model.
• test_ransac: Runs the camera localization pipeline on test images. The method requires RGB images and a CNN model. Because the method calculates evaluation metrics, ground truth poses should also be provided.

Compiling the code will require: OpenCV 2.4, PNG++, Torch and cuDNN. We compiled the code using Ubuntu 16.04 and GCC 4.9.1.

In the following, we will first give more details about the general setup and the individual programs. Then, we describe how to run the code on the datasets used in the paper. In case you have questions regarding the code, feel free to contact the first author of the associated paper.

General Setup

The DSAC++ source code is a combination of a C++ framework and multiple Torch scripts. All data access, geometric operations and accuracy evaluation is performed in C++. The Torch scripts are called from C++ to grant access to the CNNs used in the pose estimation pipeline. We use two separate Torch scripts, one for regressing scene coordinates given an RGB image, and another one regressing hypothesis scores given re-projection error images. Note that the second script does not load a learnable CNN but instead constructs a soft inlier count for easy back propagation and the implementation of our entropy control schema (see Sec. 2.3. of our paper). The Torch scripts are contained in the sub-folder core/lua. The script variants ending with nomodel are used when training our pipeline with the scene coordinate heuristic. They offer the exact same functionality as the standard scripts but store their respective CNNs under a different file name.

Setting Parameters

All C++ programs share a set of parameters which the user can specify, see below for a complete list. There are two ways of setting the parameters. The first and direct way is to append it to the program call in the command line. Every parameter has a few letter abbreviation (case sensitive, see below) which can be given to the command line starting with a dash and followed by a space and the value to set, e.g. -ih 480 -iw 640. Because there might be a large number of parameters to set, there is also the second way which is providing a default.config file. This file should contain one parameter per line. More specifically, each line should start with the parameter abbreviation (without a leading dash) followed by a space and the value to set. The config file can also include comment lines which start with the # symbol. All programs will first look for the default.config file which will be parsed at the very beginning of each run. After that, command line parameters will be processed (overwriting parameters from the default.config). Our data package (see below) contains an example default.config file for the 3 datasets used in the experiments of the paper.

Parameters related to the data:

iw width of input images (px)

ih height of input images (px)

fl focal length of the RGB camera

xs x position of the principal point of the RGB camera

ys y position of the principal point of the RGB camera

oscript Torch script file for learning scene coordinate regression

sscript Torch script file for hypothesis scoring

omodel file storing the scene coordinate regression CNN

cd constant depth prior in meters to be used for the scene coordinate heuristic (set 0 to use provided depth images)

sid arbitrary string to be attached to output files (but does not affect CNN model file names), handy to differentiate mutliple runs of the pipeline

iSS sub-sampling of training images, e.g. iSS 10 will use 10% of the training data

cSS sub-sampling of the CNN architecture output w.r.t. the input (necessary for data exchange between C++ and Torch, e.g. the paper architecture takes an 640x480 image and predicts 80x60 coordinates, i.e. cSS is 8)

Parameters related to pose estimation:

rdraw draw a hypothesis randomly (rdraw 1), i.e. DSAC, or take the one with the largest score (rdraw 0), i.e. RANSAC

rT re-projection error threshold (in px) for measuring inliers in the pose estimation pipeline

rI initial number of pose hypotheses drawn per frame

rRI number of refinement iterations

Parameters related to learning:

Parameters which concern the learning of the scene coordinate regression CNN and the soft inlier score (e.g. learning rate or file names to store the CNNs) are defined directly in the corresponding Torch scripts (see core/lua). There are also some parameters defined in the respective main cpp file of each program, such as the total number of training iterations.

Training Procedure

As stated in the paper, end-to-end training (i.e. calling train_ransac) needs a good initialization in terms of the scene coordinate regression CNN. Therefore, pre-training (i.e. calling train_obj, potentially followed by train_repro) should be executed first. An example order of executing the programs is given at further below.

Scene Coordinate Ground Truth

In order to pre-train the scene coordinate regression CNN (i.e. calling train_obj), scene coordinate ground truth has to be available for each training frame. Our code generates scene coordinate ground truth from depth frames, the ground truth poses and camera calibration parameters. Depth frames can come from a RGB-D camera, a renderer or an arbitrary depth estimation algorithm. Alignment is assumed between RGB and depth images. Alternatively, train_obj can utilize a scene coordiante heuristic based on a constant depth assumption (parameter cd in meters). In this case, the scene coordinate regression CNN should be further optimized using train_repro for good accuracy.

Programs

In the following, we describe input and output of the individual programs created by compiling the code. An example order of executing the programs is given at the end of the document.

train_obj

The program train_obj pre-trains a scene coordinate regression CNN by minimizing the L1 distance between the predicted coordinate and the ground truth coordinate. The program needs access to a Torch script which constructs and grants access to a scene coordinate regression CNN (default: train obj.lua). The program will store the current state of the CNN in a fixed interval of parameter updates. This interval and the file name is defined in the Torch script. The program creates a training loss txt file which contains per line the training iteration number followed by the training loss at that iteration (mean over an image). When no depth images are available, the cd parameter should be set to a positive value in meters (e.g. 3 for indoor and 10 for outdoor scenes). The program will then calculate ground truth scene coordinate assuming a constant depth for each image.

train_repro

The program train_repro loads a pre-trained scene coordinate regression CNN and refines it by optimizing the reprojection error of its predictions. The program needs access to a Torch script to load and access the scene coordinate regression CNN (default: train_obj.lua). The CNN to load can be specified via -omodel. The program will store the current states of both CNNs in fixed intervals of parameter updates. These intervals and the file names are defined in the Torch scripts. The program also creates a repro training loss txt file which contains the training loss per training iteration.

train_ransac

The program train_ransac loads a pre-trained scene coordinate regression CNN and refines it by training the complete camera localization pipeline end-to-end using the DSAC strategy. The program optimizes the expectation of the pose loss. The program needs access to two Torch scripts to load and access the scene coordinate regression CNN (default: train_obj_e2e.lua) and the soft inlier score (default: score_incount_ec6.lua). The CNN to load can be specified via -omodel. The program will store the current states of both CNNs in fixed intervals of parameter updates. These intervals and the file names are defined in the Torch scripts. The