AI4Earth: AI Assisted Aerial Imagery Analysis for Wildlife Population and Human Conflict in Tanzania

Mitigating wildlife and human conflicts with AI, funded by Microsoft AI for Earth Innovation Grant and Global Wildlife Conservation.

Labelling

Data labeling is done using Computer Vision Annotation Tool (CVAT) tool.

Training dataset annotation workflow using CVAT. Aerial imagery was annotated by a group of volunteers in Tanzania. The annotators created 30 classes of labels that covered wildlife, livestock, and human activities. The final labeled data is tiled/chipped and converted into TFRecords format as machine learning ready data for the coming model training.

Aerial survey and the images

Aerial surveys of large mammals in Africa are normally done from light aircraft (Cessna 172/182/206 type) at an altitude above ground level (AGL) of 90-110m (300 to 350 feet).

There are two types of images that are available from aerial surveys for this project:

RSO

During the survey, Rear seat observers (RSO), window-mounted cameras are used to verify herd size and ID - they are triggered by the human observer when he/she sees a target of interest (all elephants, and larger herds of any species). The aircaft flies straight lines (transects) back and forth over the target area.

• These images are lower quality as they are taken through the Plexiglas.
• Most images 24 megapixel (MP), taken at 35° down-angle from horizontal.
• Very target-rich set of images which will produce many examples for training. This is because a human has triggered the camera on a target.
• Arosund 20,000 images are currently available and being reviewed by the annotation lab.
• GPS metadata are often not available (camera clocks not synchronized perfectly with GPS).

New aerial survey

The new system is for cameras on the wing struts which take constant images along flight paths.

• These images have no intervening Plexiglas and are much higher quality.
• Taken at 45° down-angle.
• 24MP and optimized for image sharpness.
• Very low rate of 'positives' - perhaps 2% of images will have any wildlife or livestock present.
• Around 500,000 images are available.
• GPS metadata present for all images.

<img width="903" alt="PAS-aerial-survey" src="https://user-images.githubusercontent.com/14057932/106537752-f6b18d00-64c8-11eb-99f7-f42cbc457264.png">

Training Data Generation with the aiaiatrain.azurecr.io/aiaia_tfrecord_eval_cpu container

The aiaiatrain.azurecr.io/aiaia_tfrecord_eval_cpu container has multiple uses: generating training data, running evaluation on model outputs, and serving as an interactive jupyter notebook server or testing server with pytest.

• Build and access the container locally. This will take a long time to build depending on your internet connection. You can change the docker-compose.yaml file to have it build to support gpu, which will make evaluation faster.

docker-compose build
USERID=$(id -u) docker-compose run aiaiatrain.azurecr.io/aiaia_tfrecord_eval_cpu bash

Once in the docker container ,run the following commands 👇

TFrecords creation

• Adding label_id, category and group in the CSV files

  From: https://docs.google.com/spreadsheets/d/1zWjgRcFwZh_OfpE8bUQoBBC3Imf8Ixua4eYBurjMUG4/edit#gid=0

python3 add_class_id.py \
    --csv=TA25_train_sliced_image_nbboxes.csv \
    --csv_output=TA25_train_sliced_image_nbboxes_class_id.csv

Files were stored at: s3://aisurvey/training_data202008/P1000/*_train_sliced_image_nbboxes_class_id.csv They have been moved to the aiaiatrain container on Azure.

• Creating Tfrecords for object detection

python3 tf_records_creation.py \
    --tile_path=aisurvey/training_data202008/P1000/SL25_tiles \
    --csv=aisurvey/training_data202008/P1000/SL25_train_sliced_image_nbboxes_class_id.csv \
    --csv_class_map=aisurvey/class_map/class_map.csv \
    --output_dir=/aisurvey/training_data202008/P1000/SL25_tfrecord \
    --width=1000 \
    --height=1000

or execute the bash file:

./write_tfrecords.sh

PBTXT creation

python3 write_training_pbtxt.py \
    --csv aisurvey/class_map/class_map.csv \
    --out_dir=aisurvey/training_data202008/P1000/pbtext/

outputs at: s3://aisurvey/training_data202008/P1000/pbtxt They are also stored in the aiaiatrain container on Azure.

• Creating Tfrecords for classification

python3 tf_records_creation_classification.py \
        --tile_path=data/P400_v2/ \
        --csv_files=data/csv/*_class_id.csv \
        --output_dir=data/classification_training_tfrecords/ \
        --output_csv=data/csv/classification_training_tfrecords.csv

or execute the bash file:

./write_tfrecords_classification.sh

AI-assisted Aerial Imagery Analysis (AIAIA)

At the end of the project, we present two AI-assisted systems: 1) an image classifier, AIAIA Classifier, that filters images containing objects of interest from tens of thousands of aerial images using automated camera systems and 2) a set of three object detection models, AIAIA Detectors, which locate, classify and count objects of interest within those images. The detected objects were assigned to image IDs that have their unique geolocation recorded during the aerial surveys. These geocoded detections were then used to generate maps of the distribution of wildlife, human activities, and livestock, with a visualisation of mapped proximity highlighting potential conflict areas. Explore the map here.

<img width="1131" alt="AIAIA-workflow" src="https://user-images.githubusercontent.com/14057932/106538113-b6064380-64c9-11eb-86f4-602bc8ab41dc.png">

AIAIA Detectors

All of the following paths are relative to the gcpprocessedtraining container and can be downloaded locally, or mounted or downloaded to a VM. See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux

Building the training image

The training images have already been built and uploaded to an azure container registry named aiaiatrain. You can pull it to a VM or local machine with:

cd aiaia-detector
az acr login --name aiaiatrain # login to the azure container registry
docker run -u 0 --rm -v ./:/mnt/data -it aiaiatrain.azurecr.io/aiaia-tf1.15-frozen_graph:latest bash

If you need to build them and upload them again to ACR do the following.

To build the training image, run the following command, replacing VERSION with an appropriate value (e.g., v2):

cd aiaia_detector/
bash build_azure_images.sh
docker push

See the README.md in the aiaia_detector folder for instructions on training.

Watching model training with Tensorboard

After an object detection model is trained and model checkpoints. You can go through these few steps to visualize the tensorboard output that you can also view during azureml training. (see README.md in the azure folder):

Download the outputs in gcsprocessedtraining/azureml_outputs_detector/model_logs

Visualize tensorboard locally

tensorboard --logdir='path/to/model_logs'

Model inference

These scripts run model inference with frozen graph files (which end in .pb). We faced a latency issue running with TFserving images, so we recommend using the frozen_graph.pb files for the object detector and classifier inference.

Running the inference script in local docker container

• Getting required files, download these to the folder mounted to the container.

cd AIAIA/inference
# create folder
mkdir -p models/wildlife
mkdir -p models/livestock
mkdir -p models/human-activities
mkdir -p data/chips/
# Downloading the frozen graph model, example chips, and pbtxt file of labels from gcsprocessedtraining
azcopy cp "https://aiaiatrain.blob.core.windows.net/gcsprocessedtraining/chips/*<SAS>" ./data/chips/ --recursive=true
azcopy cp "https://aiaiatrain.blob.core.windows.net/gcsprocessedtraining/root_data_for_azureml_detector/tf1models_p400/ryan_sprint_rcnn_resnet101_serengeti_wildlife_v3_tfs/frozen_inference_graph.pb<SAS>" ./models/wildlife/ --recursive=true
#note: the spaces in these urls are not typos. space characters left over when copying from gcp
azcopy cp "https://aiaiatrain.blob.core.windows.net/gcsprocessedtraining/root_data_for_azureml_detector/tf1models_p400/ rcnn_resnet101_serengeti_livestock_image_tensor/frozen_inference_graph.pb<SAS>" ./models/livestock
azcopy cp "https://aiaiatrain.blob.core.windows.net/gcsprocessedtraining/root_data_for_azureml_detector/tf1models_p400/ rcnn_resnet101_serengeti_human_activities_image_tensor/frozen_inference_graph.pb<SAS>" ./human-activities
azcopy cp "https://aiaiatrain.blob.core.windows.net/gcsprocessedtraining/chips/<SAS>" ./ --recursive=true

docker run -u 0 --rm -v $(pwd):/mnt/data -it aiaiatrain.azurecr.io/aiaia:v1-tf1.15-frozen-graph-cpu /bin/bash

• Executing prediction

# install some missing pip modules in the container. if the containers are rebuilt you won't need to pip install these since they are in the requirements-deploy.txt now
pip install numpyencoder click joblib tqdm

python frozen_graph/frozen_pred.py \
    --frozen_inference_graph=models/wildlife/frozen_inference_graph.pb \
    --images_path=data/chips/*.jpg \
    --threshold=0.5

The output will be store in the file: data/chips/

Running on NVIDIA GPU on top of classifier result

docker pull devseeddeploy/od_frozen_graph_inference:v1-tf1.15-gpu

docker run -u 0 --rm -v $(pwd):/mnt/data -it aiaiatrain.azurecr.io/aiaia:v1-tf1.15-frozen-graph-gpu /bin/bash

Filter image chips by the threshold score

This is optional if you have have result from an image classifier and only want to filter images that contains interested objects. The script was run on amazon aws and can be