Inference Helper

This is a wrapper of deep learning frameworks especially for inference
This class provides a common interface to use various deep learnig frameworks, so that you can use the same application code

Supported frameworks

TensorFlow Lite
TensorFlow Lite with delegate (XNNPACK, GPU, EdgeTPU, NNAPI)
TensorRT (GPU, DLA)
OpenCV(dnn) (with GPU)
OpenVINO with OpenCV (xml+bin)
ncnn (with Vulkan)
MNN (with Vulkan)
SNPE (Snapdragon Neural Processing Engine SDK (Qualcomm Neural Processing SDK for AI v1.51.0))
Arm NN
NNabla (with CUDA)
ONNX Runtime (with CUDA)
LibTorch (with CUDA)
TensorFlow (with GPU)

Overview

Supported targets

Windows 10 (Visual Studio 2019 x64)
Linux (x64, armv7, aarch64)
Android (armeabi-v7a, arm64-v8a)

CI Status

| Framework | Windows (x64) | Linux (x64) | Linux (armv7) | Linux (aarch64) | Android (aarch64) | |---------------------------|-----------------------------------------------|-----------------------------------------------|-----------------------------------------------|-----------------------------------------------|------------------------------------------------| | | | | | | | | TensorFlow Lite | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[ ] Test </li></ul> | | TensorFlow Lite + XNNPACK | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| Unsupported | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[ ] Test </li></ul> | | TensorFlow Lite + EdgeTPU | <ul><li> [x] Build</li><li>[ ] Test </li></ul>| <ul><li> [x] Build</li><li>[ ] Test </li></ul>| <ul><li> [x] Build</li><li>[ ] Test </li></ul>| <ul><li> [x] Build</li><li>[ ] Test </li></ul>| Unsupported | | TensorFlow Lite + GPU | No library | No library | No library | No library | <ul><li> [x] Build</li><li>[ ] Test </li></ul> | | TensorFlow Lite + NNAPI | Unsupported | Unsupported | Unsupported | Unsupported | <ul><li> [x] Build</li><li>[ ] Test </li></ul> | | TensorRT | <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| Unsupported | | OpenCV(dnn) | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[ ] Test </li></ul> | | OpenVINO with OpenCV | <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| Unsupported | | ncnn | <ul><li> [x] Build</li><li>[ ] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| No library | No library | <ul><li> [x] Build</li><li>[ ] Test </li></ul> | | MNN | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| No library | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[ ] Test </li></ul> | | SNPE | Unsupported | Unsupported | <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul>| <ul><li> [ ] Build</li><li>[ ] Test </li></ul> | | Arm NN | Unsupported | <ul><li> [x] Build</li><li>[x] Test </li></ul>| Unsupported | <ul><li> [x] Build</li><li>[x] Test </li></ul>| No library | | NNabla | <ul><li> [x] Build</li><li>[x] Test </li></ul>| No library | Unsupported | No library | No library | | ONNX Runtime | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| Unsupported | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[ ] Test </li></ul> | | LibTorch | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| No library | No library | No library | | TensorFlow | <ul><li> [x] Build</li><li>[x] Test </li></ul>| <ul><li> [x] Build</li><li>[x] Test </li></ul>| No library | No library | No library |

Unchedked(blank) doesn't mean that the framework is unsupported. Unchecked just means that the framework is not tested in CI. For instance, TensorRT on Windows/Linux works and I confirmed it in my PC, but can't run it in CI.
No Library means a pre-built library is not provided so that I cannot confirm it in CI. It may work if you build a library by yourself.

Sample projects

https://github.com/iwatake2222/InferenceHelper_Sample
https://github.com/iwatake2222/play_with_tflite
https://github.com/iwatake2222/play_with_tensorrt
https://github.com/iwatake2222/play_with_ncnn
https://github.com/iwatake2222/play_with_mnn

Usage

Please refer to https://github.com/iwatake2222/InferenceHelper_Sample

Installation

Add this repository into your project (Using git submodule is recommended)
Download prebuilt libraries
- sh third_party/download_prebuilt_libraries.sh

Additional steps

You need some additional steps if you use the frameworks listed below

Additional steps: OpenCV / OpenVINO

Install OpenCV or OpenVINO
- You may need to set/modify OpenCV_DIR and PATH environment variable
- To use OpenVINO, you may need to run C:\Program Files (x86)\Intel\openvino_2021\bin\setupvars.bat or source /opt/intel/openvino_2021/bin/setupvars.sh

Additional steps: TensorRT

Install CUDA + cuDNN
Install TensorRT 8.x

Additional steps: Tensorflow Lite (EdgeTPU)

Install the following library
- Linux: https://github.com/google-coral/libedgetpu/releases/download/release-grouper/edgetpu_runtime_20210726.zip
- Windows: https://github.com/google-coral/libedgetpu/releases/download/release-frogfish/edgetpu_runtime_20210119.zip
  - the latest version doesn't work
  - it may be better to delete C:\Windows\System32\edgetpu.dll to ensure the program uses our pre-built library

Additional steps: ncnn

Install Vulkan
- You need Vulkan even if you don't use it because the pre-built libraries require it. Otherwise you need to build libraries by yourself disabling Vulkan
- https://vulkan.lunarg.com/sdk/home
- Windows
  - https://sdk.lunarg.com/sdk/download/latest/windows/vulkan-sdk.exe
  - It's better to check (Optional) Debuggable Shader API Libraries -64-bit , so that you can use Debug in Visual Studio
- Linux (x64)
```
wget https://sdk.lunarg.com/sdk/download/latest/linux/vulkan-sdk.tar.gz
tar xzvf vulkan-sdk.tar.gz
export VULKAN_SDK=$(pwd)/1.2.198.1/x86_64
sudo apt install -y vulkan-utils libvulkan1 libvulkan-dev
```

Additional steps: SNPE

Download library from https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/tools
Extract snpe-1.51.0.zip , then place lib and include folders to third_party/snpe_prebuilt

Note:

Debug mode in Visual Studio doesn't work for ncnn, NNabla and LibTorch because debuggable libraries are not provided
- Debug will cause unexpected bahavior, so use Release or RelWithDebInfo
See third_party/download_prebuilt_libraries.sh and third_party/cmakes/* to check which libraries are being used. For instance, libra

InferenceHelper

Install / Use

README