---

page_type: sample languages:

• python products:
• iotedge description: "The IntelligentEdgeHOL walks through the process of deploying an IoT Edge module to an Nvidia Jetson Nano device to allow for detection of objects in YouTube videos, RTSP streams, or an attached web cam " urlFragment: "https://github.com/Azure/IntelligentEdgeHOL"

---

Introduction

The IntelligentEdgeHOL walks through the process of deploying an IoT Edge module to an Nvidia Jetson Nano device to allow for detection of objects in YouTube videos, RTSP streams, Hololens Mixed Reality Capture, or an attached web cam. It achieves performance of around 10 frames per second for most video data. Micheleen Harris has created a fork of this project which targets Jetson Xavier devices using YOLOv4 to achieve even higher performance and accuracy.

The module ships as a fully self-contained docker image totalling around 5.5GB. This image contains all necessary dependencies including the Nvidia Linux for Tegra Drivers for Jetson Nano, CUDA Toolkit, NVIDIA CUDA Deep Neural Network library (CUDNN), OpenCV, and Darknet. For details on how the base images are built, see the included docker folder.

Object Detection is accomplished using YOLOv3-tiny with Darknet which supports detection of the following:

person, bicycle, car, motorbike, aeroplane, bus, train, truck, boat, traffic light, fire hydrant, stop sign, parking meter, bench, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove,skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, chair, sofa, pottedplant, bed, diningtable, toilet, tv monitor, laptop, mouse, remote, keyboard, cell phone, microwave, oventoaster, sink, refrigerator, book, clock, vase, scissors, teddy bear, hair drier, toothbrush

Demos

• Yolo Object Detection with Nvidia Jetson and Hololens

Hands-On Lab Materials

• Presentation Deck
• Presentation Video
  • Note: If you want to view a full walkthrough of this lab, skip to 38:00

Getting Started

This lab requires that you have the following:

Hardware:

• Nvidia Jetson Nano Device
• A cooling fan installed on or pointed at the Nvidia Jetson Nano device
• USB Webcam (Optional)
  • Note: The power consumption will require that your device is configured to use a 5V/4A barrel adapter as mentioned here with an Open-CV compatible camera.

Development Environment:

• Visual Studio Code (VSCode)
• VSCode Extensions
  • Azure Account Extension
  • Azure IoT Edge Extension
  • Docker Extension
  • Azure IoT Toolkit Extension
• Git tool(s)
  Git command line

Installing IoT Edge onto the Jetson Nano Device

Before we install IoT Edge, we need to install a few utilities onto the Nvidia Jetson Nano device with:

sudo apt-get install -y curl nano python3-pip

ARM64 builds of IoT Edge are currently being offered in preview and will eventually go into General Availability. We will make use of the ARM64 builds to ensure that we get the best performance out of our IoT Edge solution.

These builds are provided starting in the 1.0.8 release tag. To install the 1.0.8 release of IoT Edge, run the following from a terminal on your Nvidia Jetson device:

# You can copy the entire text from this code block and 
# paste in terminal. The comment lines will be ignored.

# Install the IoT Edge repository configuration
curl https://packages.microsoft.com/config/ubuntu/18.04/multiarch/prod.list > ./microsoft-prod.list

# Copy the generated list
sudo cp ./microsoft-prod.list /etc/apt/sources.list.d/

# Install the Microsoft GPG public key
curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo cp ./microsoft.gpg /etc/apt/trusted.gpg.d/

# Perform apt update
sudo apt-get update

# Install IoT Edge and the Security Daemon
sudo apt-get install iotedge

Provisioning the IoT Edge Runtime on the Jetson Nano Device

To manually provision a device, you need to provide it with a device connection string that you can create by registering a new IoT Edge device in your IoT hub. You can create a new device connection string to accomplish this by following the documentation for Registering an IoT Edge device in the Azure Portal or by Registering an IoT Edge device with the Azure-CLI.

Once you have obtained a connection string, open the configuration file:

sudo nano /etc/iotedge/config.yaml

Find the provisioning section of the file and uncomment the manual provisioning mode. Update the value of device_connection_string with the connection string from your IoT Edge device.

provisioning:
  source: "manual"
  device_connection_string: "<ADD DEVICE CONNECTION STRING HERE>"
  
# provisioning: 
#   source: "dps"
#   global_endpoint: "https://global.azure-devices-provisioning.net"
#   scope_id: "{scope_id}"
#   registration_id: "{registration_id}"

After you have updated the value of device_connection_string, restart the iotedge service with:

sudo service iotedge restart

You can check the status of the IoT Edge Daemon using:

systemctl status iotedge

Examine daemon logs using:

journalctl -u iotedge --no-pager --no-full

And, list running modules with:

sudo iotedge list

Configuring the YoloModule Video Source

Clone or download a copy of this repo and open the IntelligentEdgeHOL folder in Visual Studio Code. Next, press F1 and select Azure IoT Hub: Select IoT Hub. Next, choose the IoT Hub you created when provisioning the IoT Edge Runtime on the Jetson Nano Device and follow the prompts to complete the process.

In VS Code, navigate to the .env file and modify the following value:

CONTAINER_VIDEO_SOURCE

To use a youtube video, provide a Youtube URL, ex: https://www.youtube.com/watch?v=YZkp0qBBmpw

For an rtsp stream, provide a link to the rtsp stream in the format, rtsp://

To use a HoloLens video stream, see this article to enable a user account in the HoloLens Web Portal, once this is configured, provide the url to the HoloLens video streaming endpoint, ex: https://[USERNAME]:[PASSWORD]@[HOLOLENS_IP]/api/holographic/stream/live_high.mp4?holo=true&pv=true&mic=true&loopback=true

If you have an attached USB web cam, provide the V4L device path (this can be obtained from the terminal with ls -ltrh /dev/video*, ex: /dev/video0 and open the included deployment.template.json and look for:

{
   "PathOnHost": "/dev/tegra_dc_ctrl",
   "PathInContainer":"/dev/tegra_dc_ctrl",
   "CgroupPermissions":"rwm"                      
}

Then, add the following (including the comma), directly beneath it

,
{
   "PathOnHost": "/dev/video0",
   "PathInContainer":"/dev/video0",
   "CgroupPermissions":"rwm"                      
}

Deploy the YoloModule to the Jetson Nano device

Create a deployment for the Jetson Nano device by right-clicking deployment.template.json and select Generate IoT Edge Deployment Manifest. This will create a file under the config folder named deployment.arm32v7.json, right-click that file and select Create Deployment for Single Device and select the device you created when provisioning the IoT Edge Runtime on the Jetson Nano Device.

It may take a few minutes for the module to begin running on the device as it needs to pull an approximately 5.5GB docker image. You can check the progress on the Nvidia Jetson device by monitoring the iotedge agent logs with:

sudo docker logs -f edgeAgent

Example output:

2019-05-15 01:34:09.314 +00:00 [INF] - Executing command: "Command Group: (
  [Create module YoloModule]
  [Start module YoloModule]
)"
2019-05-15 01:34:09.314 +00:00 [INF] - Executing command: "Create module YoloModule"
2019-05-15 01:34:09.886 +00:00 [INF] - Executing command: "Start module YoloModule"
2019-05-15 01:34:10.356 +00:00 [INF] - Plan execution ended for