STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Dataset

Meida Chen, Qingyong Hu, Zifan Yu, Hugues Thomas, Andrew Feng, Yu Hou, Kyle McCullough, Fengbo Ren, Lucio Soibelman. [Project page] [Paper] [BMVC presentation] [Demo video] [Poster] [Urban3D workshop@ICCV2023][Urban3D workshop@ECCV2022] [Instance segmentation competition]

Updates

03/31/2023: we are organizing the Urban3D@ICCV2023 - The 3rd Challenge on Large-Scale Point Clouds Analysis for Urban Scenes Understanding!
11/29/2022: All source images are released Here.
10/23/2022: Congrats to our Urban3D team for successfully organizing the Urban3D workshop at ECCV 2022; Over 300 teams participated and competed on the SensatUrban (semantic segmentation) and STPLS3D (instance segmentation) datasets. All winners surpassed our baseline methods by a large margin. Replay of the workshop video is available.
10/14/2022: Special thanks to Jonas Schult for implementing Mask3D for STPLS3D - instance segmentation! Please refer to the official Mask3D for implementation details, and download their pretrained model.
10/13/2022: Our Paper is accepted as oral presentation at BMVC2022!
06/28/2022: Special thanks to Thang Vu for implementing SoftGroup for STPLS3D - instance segmentation! Please refer to the official SoftGroup for implementation details, and download their pretrained model.
03/25/2022: we are organizing the Urban3D@ECCV2022 - The 2nd Challenge on Large-Scale Point Clouds Analysis for Urban Scenes Understanding!
11/01/2021: Initial release!

(1) Our Focus

Our project aims to provide a large database of annotated ground truth point clouds reconstructed using aerial photogrammetry.
Our database can be used for training and validating 3D semantic and instance segmentation algorithms.
We are developing a synthetic data generation pipeline to create synthetic training data that can augment or even replace real-world training data.

(2) Dataset

2.1 Download

To download the STPLS3D point clouds for Semantic Segmentation click Here.
To download the STPLS3D point clouds for Instance Segmentation click Here.
To download the unlabled testing datasets for STPLS3D instance segmentation competition click Here.
To download the Source Images for both synthetic and real-world datasets click Here.

2.2 Overview

we have built a large-scale photogrammetry 3D point cloud dataset, termed Semantic Terrain Points Labeling - Synthetic 3D (STPLS3D), which is composed of high-quality, rich-annotated point clouds from real-world and synthetic environments.

2.3 Data Collection

We first collect real-world aerial images using photogrammetry best practices with quadcopter drone flight at a low altitude with significant overlaps between adjacent photos. We then reconstructed point clouds with 1.27 km^2 landscape following the standard photogrammetry pipeline. Next, we follow the same UAV path and flying pattern to generate 62 synthetic point clouds with different architectural styles, vegetation types, and terrain shapes. The synthetic dataset covers about 16 km^2 of the city landscape, with up to 18 fine-grained semantic classes and 14 instance classes.

2.4 Synthetic data generation workflow demo

2.5 Semantic Annotations

0-Ground: including grass, paved road, dirt, etc.
1-Building: including commercial, residential, educational buildings.
2-LowVegetation: 0.5 m < vegetation height < 2.0 m.
3-MediumVegetation: 2.0 m < vegetation height < 5.0 m.
4-HighVegetation: 5.0 m < vegetation height.
5-Vehicle: including sedans and hatchback cars.
6-Truck: including pickup trucks, cement trucks, flat-bed trailers, trailer trucks, etc.
7-Aircraft: including helicopters and airplanes.
8-MilitaryVehicle: including tanks and Humvees.
9-Bike: bicycles.
10-Motorcycle: motorcycles.
11-LightPole: including light poles and traffic lights.
12-StreetSgin: including road signs erected at the side of roads.
13-Clutter: including city furniture, construction equipment, barricades, and other 3D shapes.
14-Fence: including timber, brick, concrete, metal fences.
15-Road: including asphalt and concrete roads.
17-Windows: glass windows.
18-Dirt: bare earth.
19-Grass: including grass lawn, wild grass, etc.

Note that not all datasets we are currently providing have all the semantic labels available, the ground points that don't have the material available (15, 18, 19) are labeled with 0.

2.6 Instance annotations

The ground is labeled with -100. Window instance is currently per building but not per window but could be post-processed using connect component algorithm. Our experiments did not include the window instances.

Only synthetic datasets v2 and v3 have the instance labels.

(3) Benchmarks

3.1 Semantic segmentation:

3.2 Instance segmentation:

(4) Training and Evaluation

Here we provide the training and evaluation script for both semantic and instance segmentation.

4.1 Semantic segmentation:

KpConv (Ubuntu and Windows 10): The environment setup is the same as the official KpConv release. We follow the same steps as shown here to evaluate KpConv on our STPLS3D dataset.

Preparing the dataset

Download the data and unzip it. Change the variable self.path of STPLS3DDataset class (here) to the place where STPLS3D is stored.

STPLS3D
├── RealWorldData
│   ├── OCCC_points.ply
│   ├── ...
│   └── WMSC_points.ply
├── Synthetic_v1
│   ├── Austin.ply
│   ├── ...
│   └── TownshipofWashington.ply
├── Synthetic_v2
│   ├── 2_points_GTv2.ply
│   ├── ...
│   └── j_points_GTv2.ply
└── Synthetic_v3
    ├── 1_points_GTv3.ply
    ├── ...
    └── 25_points_GTv3.ply

Start training:

python3 train_STPLS3D.py

Evaluation:

python3 test_models.py

Point Transformer (Ubuntu): Please refer to Point Transformer to test it on our STPLS3D dataset.

4.2 Instance segmentation:

Mask3D: Special thanks to Jonas Schult for implementing Mask3D for STPLS3D! Please refer to the official Mask3D for implementation details, and download their pretrained model.

SoftGroup: Special thanks to Thang Vu for implementing SoftGroup for STPLS3D! Please refer to the official SoftGroup for implementation details, and downloading their pretrained model.

HAIS (Ubuntu): The environment setup is the same as the official HAIS release

Setup the environment

git clone https://github.com/meidachen/STPLS3D.git
cd STPLS3D/HAIS
conda create -n hais python=3.7
conda activate hais
pip install -r requirements.txt
conda install -c bioconda google-sparsehash
conda install libboost
conda install -c daleydeng gcc-5
cd STPLS3D/HAIS/lib/spconv
export CUDACXX= $PATH_TO_NVCC$
python setup.py bdist_wheel
cd STPLS3D/HAIS/lib/spconv/dist
pip install {wheel_file_name}.whl
cd STPLS

STPLS3D

Install / Use

README