ObjectTracking

<h1>Object Tracking BrickPi Robot</h1><br> This page is about a project to create an intelligent Raspberry Pi powered mobile robot. The goal is to have a robot that can teach itself to track and chase objects. There are three phases: Object Tracking, Motor Control and Machine Learning. The first two are complete and the robot can track an object of a given colour and chase it. However it does not teach itself to do this. Over the summer one of my students will be creating an <a href="http://en.wikipedia.org/wiki/Artificial_neural_network">Artificial Neural Network</a> for the machine learning part of the project. <h1>Phase 1: Object detection</h1>

[]

There are two version of the files: one for use on a Raspberry Pi with camera module, and the other for use on a PC with webcam. There are few changes, but in order to keep the code clean and clear I have made separate versions.

<p>First you need to make sure that your Raspberry Pi is properly set up and can obtain an image programmatically. My preferred option is to use v4l2. You will also need to install the open Computer Vision library <a href="http://docs.opencv.org">openCV</a>, with 'sudo apt-get install libopencv-dev python-opencv'.</p> <h2>1. Getting an image</h2> <p>Once you have those elements installed you can obtain an openCV image object with the python program <a href="https://github.com/MagnusBordewich/ObjectTracking/blob/master/RaspberryPi/1-capture_image.py">1-capture_image.py</a>. Open it in IDLE, and select Run>Run Module from the menu bar. You should see an image from the camera appear on screen. The press a key to see a transformed image.</p> <p> Once you have that working try the following:</p> <ul> <li>Reorienting the image if it is upside down: find the function cv2.flip(image,-1). The number in the brackets controls what sort of flip is done. Try changing it to 0 or 1, to get a correct orientation for your image, then try other numbers to see the effect.</li> <li>Obtaining images of different sizes: change the values of variables w and h in the lines "w=480" and "h=320".</li> <li>Adjust the blur, and try other <a href="http://docs.opencv.org/modules/imgproc/doc/filtering.html">image filtering</a> options.</li> </ul> <h2>2. Identify a region by hue</h2> <p>Computers normally store an image as a giant matrix with three values for each pixel: the intensity if red, green and blue (RGB values) that combine to make the colour of the pixel. A simple but fairly robust method of indetifying an object is by colour. However you want to specify the colour in a way that isn't too much affected by how light or dark the lighting on the object is, or how washed out or exposed the image is. This is tricky when specifying ranges of RGB values, but can be done by looking at the hue of the object.</p> <div style="float:right" > <img src="images/HSV.png" height="180"/> </div> <p> This is done in the program <a href="https://github.com/MagnusBordewich/ObjectTracking/blob/master/RaspberryPi/2-detect_hue.py">2-detect_hue.py</a>, again press a key to step through the images. The function cv2.cvtColor(image,cv2.COLOR_BGR2HSV) converts the representation from three RGB values for each pixel, to a Hue, Saturation and Value value for each pixel. Hue give the essential colour, Saturation gives the intensity of that colour and Value gives the overall brightness of the pixel, as depicted in this image.</p> <p> By specifying a tight range of hue values, and a very wide range of saturation and value values, we should identify all regions that contain objects of a given colour, regardless of lighting conditions. The print statement in the program will output the HSV values of the centre pixel of the image to the console.</p> <p> The variables lower_pink and upper_pink in the program are used to specify Hue between 160 and 175, which is roughly the pink of pink post-it notes, and saturation and value values between 50 and 255, i.e. weak and dark up to strong and bright pink.</p> <p> The function cv2.inRange is used to create a mask - a matrix of 0s and 255s with 255s where the corresponding pixel was sufficiently pink, and a 0 elsewhere. I also create an opposite mask (mask_inverted), by swapping 0s and 255s. 0 and 255 are used, because when interpreted as a greyscale image, this gives a black and white mask. The masks are used to make two images - one where I convert the original image to greyscale, then do a bitwise-and with the inverted mask to keep only pixels that were not pink, and the other from a bitwise-and of the original image and the mask to keep only the pink pixels. Combining these gives an image where pink parts are kept but everything else is greyscale. </p> <p> Once you have this working try the following:</p> <ul> <li>The lines "cv2.imshow('View',xxx)" and "cv2.waitKey(0)" show the image 'xxx' and wait for a key press before continuing. Try putting in extra copies of these lines at different points, changing xxx to mask, mask_inverted or image_masked, to see steps of the process.</li> <li>Adjusting the HSV values in lower_pink and upper_pink to target a completely different colour.</li> <li>If you can't work out the HSV values of something you wish to target, then use the program <a href="https://github.com/MagnusBordewich/ObjectTracking/blob/master/RaspberryPi/HSV_finder.py">HSV_finder.py</a> which will show live the HSV value in the centre of the image.</li> </ul> <h2>3. Target the direction with the greatest match for your hue</h2> <p>The next step, program <a href="https://github.com/MagnusBordewich/ObjectTracking/blob/master/RaspberryPi/3-add_line.py">3-add_line.py</a>, involves a basic operation on the matrix values and a loop. First we take the mask, which is a matrix of 255s and 0s, where 255s represent pixels that are pink. We want the total number of pink pixels in each vertical line of the image. </p> <p>We can get this from the function np.sum(mask,axis=0). (Note: np. means use a <a href="http://www.numpy.org">NumPy</a> operation, which is a useful library of mathemtical functions in Python.) np.sum(mask,axis=0) means take the matrix 'mask' and sum each column, so LRarray = np.sum(mask,axis=0)/255 means we set LRarray to be a list of numbers, one for each column, where each value is the sum of the column values divided by 255 - i.e. the number of pink pixels in that vertical line.</p> <p> The loop starts with 'for i in range(w):'. This means 'for each value of i in the range 0 up to to w-1 do whatever is in the indented lines below'. Computers always count from 0, so we cover each of the w columns by counting from 0 to w-1. Python programs use indentation to define loops and other blocks of code - so getting the indentation right is important.</p> <p> In the loop, for each i, we look at the ith entry in our list of numbers LRarray (denoted LRaray[i]) and ask if it is the biggest we have seen yet. The biggest value seen yet is called max_x_intensity; if LRarray[i] is bigger than this, then we remember this value of i (by setting max_x_coordinate to i), and reset max_x_intensity to the new biggest, otherwise we skip those two lines.</p> <p> The result of the loop is that we know what value of i corresponds to the column with the largest number of pink pixels, and we have remembered it as max_x_coordinate. We then draw a line on the image from the top of this column to the bottom.</p> <p> Once you are happy with this, try the following:</p> <div style="float:right" > <img src="images/Daffodils.jpg" height="180"/> </div> <ul> <li>Change the axis of the np.sum to 1, so that you sum over rows instead of columns. You will have to loop over a range h, instead of w, since there are h rows in the image. Then change the line drawn to be horizontal through the most pink row by changing the coordinates in the cv2.line function. </li> <li>Or add both horizontal and vertical lines and see if you pick out the middle of the pink object.</li> <li>Rather than adding only a line at the most pink point, try adding a line at the bottom of every column with height proportional to the number of pink pixels in that column (i.e. a histogram, done here for isolating yellow).</li> <li>Consider how you might use this type of information to identify objects, or work out the size of pink objects.</li> </ul> <h2>4. Do it for real-time video</h2> Now that we have loops, it is an easy step to do everything for live video, as is done in program <a href="https://github.com/MagnusBordewich/ObjectTracking/blob/master/RaspberryPi/4-video.py">4-video.py</a>. We simple add a huge loop around the code, and remove any wait-for-key-presses or showing of non-finished images. Note that we have had to indent all the previous code, so that it is in the block that will be looped over. <p>Here I used a 'while(True):' loop. This loop will just keep on looping until we break it. I have replaced the function cv2.waitKey(0), which means wait here forever for a key press, with 'key_pressed = cv2.waitKey(1)' which means only wait 1 millisecond, but if a key is pressed, remember which one in the variable key_pressed. We can then check if the Esc key was pressed (key 27) and if so break (exit the loop), if not just keep looping.</p> <ul> <li>Try putting a loop around your code from (2) or (3) to show the hue isolation live.</li> </ul> <h2>5. Move from direction with greatest hue, to identifying objects</h2> <p> The approach we have so far could be confused between a large number of small pink objects one above the other and a single large pink object. Really we want to identify a single large object, and openCV has a convenient way to do this. In order to save ourselves a fair amount of effort, let's use openCV functions from here. We can use a function to find '<a href="http://opencvpython.blogspot.co.uk/2012/06/hi-this-article-is-tutorial-which-try.html">contours</a>' wh