ACE0 (ACE Zero)

This repository contains the code associated to the ACE0 paper:

Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer

Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cavallari, Áron Monszpart, Daniyar Turmukhambetov, and Victor Adrian Prisacariu

ECCV 2024, Oral

For further information please visit:

• Project page (with a method overview and videos)
• arXiv

Change Log

Note: We try to make sure all code changes are backwards compatible, i.e. the results of the ECCV 2024 paper remain reproducible. But just in case, we added a tag eccv_2024_checkpoint that you can check out to get the exact code version used for the ECCV 2024 paper.

• 2025-Nov-05: Added instructions for running standard ACE (CVPR 2023) with this codebase.
• 2025-Nov-06: Added capabilities of Scene Coordinate Reconstruction Priors (ICCV 2025), disabled by default.
  • RGB-D version of ACE/ACE0
  • Reconstruction priors: Depth distribution prior and 3D point cloud diffusion prior

Table of Contents

• Installation
• Usage
  • Basic Usage
  • Visualization Capabilities
  • Advanced Use Cases
    • Refine Existing Poses
    • Start From a Partial Reconstruction
    • RGB-D Reconstruction ("SCR Priors" paper, ICCV 2025)
    • Using Reconstruction Priors ("SCR Priors" paper, ICCV 2025)
    • Standard Relocalization ("ACE" paper, CVPR 2023)
    • Self-Supervised Relocalization
    • Train NeRF models or Gaussian splats
  • Utility Scripts
• Benchmark
  • Nerfacto
  • Splatfacto
• Paper Experiments
  • 7-Scenes
  • Mip-NeRF 360
  • Tanks and Temples
• Frequently Asked Questions
• References

Installation

This code uses PyTorch and has been tested on Ubuntu 20.04 with a V100 Nvidia GPU, although it should reasonably run with other Linux distributions and GPUs as well. Note our FAQ if you want to run ACE0 on GPUs with less memory.

We provide a pre-configured conda environment containing all required dependencies necessary to run our code. You can re-create and activate the environment with:

conda env create -f environment.yml
conda activate ace0

All the following commands in this file need to run from the repository root and in the ace0 environment.

ACE0 represents a scene using an ACE scene coordinate regression model. In order to register cameras to the scene, it relies on the RANSAC implementation of the DSAC* paper (Brachmann and Rother, TPAMI 2021), which is written in C++. As such, you need to build and install the C++/Python bindings of those functions. You can do this with:

cd dsacstar
python setup.py install
cd ..

Having done the steps above, you are ready to experiment with ACE0!

Important note: the first time you run ACE0, the script may ask you to confirm that you are happy to download the ZoeDepth depth estimation code and its pretrained weights from GitHub. See this link for its license and details. ACE0 uses that model to estimate the depth for the seed images. It can be replaced, please see the FAQ section below for details.

Docker

If you preferred to run ACE0 in a docker container, you can start it with:

docker-compose up -d 

You can then shell into the container with the following command:

docker exec -it acezero /bin/bash

From there you can follow the Gaussian Splatting tutorial described at the bottom of the README here. Make sure to add your images to the volume defined in docker-compose.yml

Usage

We explain how to run ACE0 to reconstruct images from scratch, with and without knowledge about the image intrinsics. We also explain how to use ACE0 to refine existing poses, or to initialise reconstruction with a subset of poses. Furthermore, we cover the visualization capabilities of ACE0, including export of the reconstruction as a video and as 3D models.

Basic Usage

In the minimal case, you can run ACE0 on a set of images as defined by a glob pattern.

# running on a set of images with default parameters
python ace_zero.py "/path/to/some/images/*.jpg" result_folder

Note the quotes around the glob pattern to ensure it is passed to the ACE0 script rather than being expanded by the shell.

If you want to run ACE0 on a video, you can extract frames from the video and run ACE0 on the extracted frames, see our Utility Scripts.

The ACE0 script will call ACE training (train_ace.py) and camera registration (register_mapping.py) in a loop until all images have been registered to the scene representation, or there is no change between iterations.

The result of an ACE0 reconstruction is the poses_final.txt in the result folder. These files contain the estimated image poses in the following format:

filename qw qx qy qz x y z focal_length confidence

filename is the image file relative to the repository root. qw qx qy qz is the camera rotation as a quaternion, and x y z is the camera translation. Camera poses are world-to-camera transformations, using the OpenCV camera convention. focal_length is the focal length estimated by ACE0 or set externally (see below). confidence is the reliability of an estimate. If the confidence is less than 1000, it should be treated as unreliable and possibly ignored.

The pose files can be used e.g. to train a Nerfacto or Splatfacto model, using our benchmarking scripts, see Benchmarking. Our benchmarking scripts also allow you to only convert our pose files to the format required by Nerfstudio, without running the benchmark itself.

<details> <summary>Other content of the result folder explained.</summary>

The result folder will contain files such as the following:

• iterationX.pt: The ACE scene model (the MLP network) at iteration X. Output of train_ace.py in iteration X.
• iterationX.txt: Training statistics of the ACE model at iteration X, e.g. loss values, pose statistics, etc. See ace_trainer.py. Output of train_ace.py in iteration X.
• poses_iterationX_preliminary.txt: Poses of cameras after the mapping iteration but before relocalization. Contains poses refined by the MLP, rather than poses re-estimated by RANSAC. Output of train_ace.py in iteration X.
• poses_iterationX.txt: Final poses of iteration X, after relocalization, i.e. re-estimated by RANSAC. Output of register_mapping.py in iteration X.
• poses_final.txt: The final poses of the images in the scene. Corresponds to the poses of the last relocalisation iteration, i.e. the output of the last register_mapping.py call.
• pc_final.ply: An ACE0 point cloud of the scene, for visualisation or initialisation of Gaussian splats. This output is optional and triggered using the --export_point_cloud True option of ace_zero.py.

</details>

Setting Calibration Parameters

Using default parameters, ACE0 will estimate the focal length of the images, starting from a heuristic value (70% of the image diagonal.) If you have a better estimate of the focal length, you can provide it as an initialisation parameter.

# running ACE0 with an initial guess for the focal length
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --use_external_focal_length <focal_length>

Using the call above, ACE0 will refine the focal length throughout the reconstruction process. If you are confident that your focal length value is correct, you can disable focal length refinement.

# running ACE0 with a fixed focal length
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --use_external_focal_length <focal_length> --refine_calibration False

Note: The current implementation of ACE0 supports only a single focal length value shared by all images. ACE0 currently also does assume that the principal point is at the image center, and pixels are square and unskewed. Changing these assumptions should be possible, but requires some implementation effort.

Visualization Capabilities

ACE0 can visualize the reconstruction process as a video.

# running ACE0 with visualisation enabled
python ace_zero.py "/path/to/some/images/*.jpg" result_folder --render_visualization True

With visualisation enabled, ACE0 will render individual frames in a subfolder renderings and call ffmpeg at the end. The visualisation will be saved as a video in the results folder, named reconstruction.mp4.

<details> <summary>Other content of the renderings folder explained.</summary>

• frame_N.png: The Nth frame of the video.
• iterationX_mapping.pkl: The visualisation buffer of the mapping call in iteration X. It stores the 3D point cloud of th