(PAZ) Perception for Autonomous Systems

Hierarchical perception library in Python.

Selected examples:

PAZ is used in the following examples (links to real-time demos and training scripts):

| Probabilistic 2D keypoints| 6D head-pose estimation | Object detection| |---------------------------|--------------------------| ------------------| |<img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/probabilistic_keypoints.png" width="425"> | <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/head_pose.png" width="440">| <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/object_detection.png" width="430">|

| Emotion classifier | 2D keypoint estimation | Mask-RCNN (in-progress) | |---------------------------|--------------------------| -----------------------| |<img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/emotion.gif" width="250">| <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/keypoints.png" width="410">| <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/mask.png" width="400">|

|Semantic segmentation | Hand pose estimation | 2D Human pose estimation | |---------------------------|-----------------------|-----------------| | <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/semantic_segmentation.png" width="325">| <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/minimal_hand_example.jpg" width="330"> |<img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/human_pose.gif" width="250"> |

| 3D keypoint discovery | Hand closure detection | 6D pose estimation | |---------------------------|-----------------------| --------------------------| |<img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/discovery_keypoints.png" width="335"> | <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/hand_closure_detection.gif" width="250">| <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/pix2pose_example.jpg" width="330"> |

| Implicit orientation | Attention (STNs) | Haar Cascade detector | |---------------------------|-----------------------|-----------------| |<img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/implicit_pose.png" width="335">| <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/attention.png" width="340"> | <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/haar_cascades.png" width="330"> |

| Eigenfaces |Prototypical Networks | 3D Human pose estimation | |---------------------------|-----------------------|-----------------| |<img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/eigenfaces.png" width="325">| <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/protonets.png" width="330"> | <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/human_pose3D.gif" width="250"> |

|MAML| | | |---------------------------|-----------------------|-----------------| |<img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/maml.png" width="325"> | <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/your_example_here.png" width="330">| <img src="https://raw.githubusercontent.com/oarriaga/altamira-data/master/images/blank.png" width="330"> |

All models can be re-trained with your own data (except for Mask-RCNN, we are working on it here).

Examples
Installation
Documentation
Hierarchical APIs
- High-level | Mid-level | Low-level
Additional functionality
- Implemented models
Motivation
Citation
Funding

Installation

PAZ has only three dependencies: Tensorflow2.0, OpenCV and NumPy.

To install PAZ with pypi run:

pip install pypaz --user

Documentation

Full documentation can be found https://oarriaga.github.io/paz/.

Hierarchical APIs

PAZ can be used with three different API levels which are there to be helpful for the user's specific application.

High-level

Easy out-of-the-box prediction. For example, for detecting objects we can call the following pipeline:

from paz.applications import SSD512COCO

detect = SSD512COCO()

# apply directly to an image (numpy-array)
inferences = detect(image)

There are multiple high-level functions a.k.a. pipelines already implemented in PAZ here. Those functions are build using our mid-level API described now below.

Mid-level

While the high-level API is useful for quick applications, it might not be flexible enough for your specific purpose. Therefore, in PAZ we can build high-level functions using our a mid-level API.

Mid-level: Sequential

If your function is sequential you can construct a sequential function using SequentialProcessor. In the example below we create a data-augmentation pipeline:

from paz.abstract import SequentialProcessor
from paz import processors as pr

augment = SequentialProcessor()
augment.add(pr.RandomContrast())
augment.add(pr.RandomBrightness())
augment.add(pr.RandomSaturation())
augment.add(pr.RandomHue())

# you can now use this now as a normal function
image = augment(image)

You can also add any function not only those found in processors. For example we can pass a numpy function to our original data-augmentation pipeline:

augment.add(np.mean)

There are multiple functions a.k.a. Processors already implemented in PAZ here.

Using these processors we can build more complex pipelines e.g. data augmentation for object detection: pr.AugmentDetection

Mid-level: Explicit

Non-sequential pipelines can be also build by abstracting Processor. In the example below we build a emotion classifier from scratch using our high-level and mid-level functions.

from paz.applications import HaarCascadeFrontalFace, MiniXceptionFER
import paz.processors as pr

class EmotionDetector(pr.Processor):
    def __init__(self):
        super(EmotionDetector, self).__init__()
        self.detect = HaarCascadeFrontalFace(draw=False)
        self.crop = pr.CropBoxes2D()
        self.classify = MiniXceptionFER()
        self.draw = pr.DrawBoxes2D(self.classify.class_names)

    def call(self, image):
        boxes2D = self.detect(image)['boxes2D']
        cropped_images = self.crop(image, boxes2D)
        for cropped_image, box2D in zip(cropped_images, boxes2D):
            box2D.class_name = self.classify(cropped_image)['class_name']
        return self.draw(image, boxes2D)
        
detect = EmotionDetector()
# you can now apply it to an image (numpy array)
predictions = detect(image)

Processors allow

Paz

Install / Use

README