DeepBeliefSDK

The SDK for Jetpac's iOS, Android, Linux, and OS X Deep Belief image recognition framework.

This is a framework implementing the convolutional neural network architecture described by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. The processing code has been highly optimized to run within the memory and processing constraints of modern mobile devices, and can analyze an image in under 300ms on an iPhone 5S. It's also easy to use together with OpenCV.

We're releasing this framework because we're excited by the power of this approach for general image recognition, especially when it can run locally on low-power devices. It gives your phone the ability to see, and I can't wait to see what applications that helps you build.

Getting started

• iOS
• Android
• Linux
• OS X
• Raspberry Pi 1
• Raspberry Pi 2
• Beaglebone Black
• Javascript
• Building from Source

Adding to an existing application

• iOS
• Android
• Linux
• OS X
• Using with OpenCV

Documentation

• Examples
• Networks
• API Reference
• FAQ
• More Information
• License
• Credits

Getting Started on iOS

You'll need the usual tools required for developing iOS applications - XCode 5, an OS X machine and a modern iOS device (it's been tested as far back as the original iPhone 4). Open up the SimpleExample/SimpleExample.xcodeproj, build and run.

You should see some warnings (the example is based on Apple sample code which has some anachronisms in it unfortunately), then once it's running a live camera stream should be visible on your phone. Move it to look closely at your keyboard, and some tags should start appearing in the top left of the screen. These should include things that look like keyboards, including calculators, remote controls, and even typewriters!

You should experiment with other objects like coffee cups, doors, televisions, and even dogs if you have any handy! The results will not be human quality, but the important part is that they're capturing meaningful attributes of the images. Understanding images with no context is extremely hard, and while this approach is a massive step forward compared to the previous state of the art, you'll still need to adapt it to the domain you're working in to get the best results in a real application.

Happily the framework includes the ability to retrain the network for custom objects that you care about. If you have logos you need to pick out, machine parts you need to spot, or just want to be able to distinguish between different kinds of scenes like offices, beaches, mountains or forests, you should look at the LearningExample sample code. It builds a custom layer on top of the basic neural network that responds to images you've trained it on, and allows you to embed the functionality in your own application easily.

There's also this full how-to guide on training and embedding your own custom object recognition code.

Adding to an existing iOS application

To use the library in your own application:

1. Add the following frameworks to the Link Binary with Libraries in your XCode project's Build Phases:
2. DeepBelief.framework
3. Accelerate.framework
4. libc++.dylib
5. Add #import <DeepBelief/DeepBelief.h> to the top of the file you want to use the code in.

You should then be able to use code like this to classify a single image that you've included as a resource in your bundle. The code assumes it's called 'dog.jpg', but you should change it to match the name of your file.

  NSString* networkPath = [[NSBundle mainBundle] pathForResource:@"jetpac" ofType:@"ntwk"];
  if (networkPath == NULL) {
    fprintf(stderr, "Couldn't find the neural network parameters file - did you add it as a resource to your application?\n");
    assert(false);
  }
  network = jpcnn_create_network([networkPath UTF8String]);
  assert(network != NULL);

  NSString* imagePath = [[NSBundle mainBundle] pathForResource:@"dog" ofType:@"jpg"];
  void* inputImage = jpcnn_create_image_buffer_from_file([imagePath UTF8String]);

  float* predictions;
  int predictionsLength;
  char** predictionsLabels;
  int predictionsLabelsLength;
  jpcnn_classify_image(network, inputImage, 0, 0, &predictions, &predictionsLength, &predictionsLabels, &predictionsLabelsLength);

  jpcnn_destroy_image_buffer(inputImage);

  for (int index = 0; index < predictionsLength; index += 1) {
    const float predictionValue = predictions[index];
    char* label = predictionsLabels[index % predictionsLabelsLength];
    NSString* predictionLine = [NSString stringWithFormat: @"%s - %0.2f\n", label, predictionValue];
    NSLog(@"%@", predictionLine);
  }
  
  jpcnn_destroy_network(network);

Getting Started on Android

I've been using Google's ADT toolchain. To get started import the AndroidExample into their custom version of Eclipse, build and run it. Hopefully you should see a similar result to the iPhone app, with live video and tags displayed. You'll need to hold the phone in landscape orientation, look for the tag text and use that as your guide.

The Android implementation uses NEON SIMD instructions, so it may not work on older phones, and will definitely not work on non-ARM devices. As a benchmark for expected performance, classification takes around 650ms on a Samsung Galaxy S5.

Adding to an existing Android application

Under the hood the Android implementation uses a native C++ library that's linked to Java applications using JNA. That means the process of including the code is a bit more complex than on iOS. If you look at the AndroidExample sample code, you'll see a 'libs' folder. This contains a deepbelief.jar file that has the Java interface to the underlying native code, and then inside the armeabi there's jnidispatch.so which is part of JNA and handles the mechanics of calling native functions, and libjpcnn.so which implements the actual object recognition algorithm. You'll need to replicate this folder structure and copy the files to your own application's source tree.

Once you've done that, you should be able to import the Java interface to the library:

import com.jetpac.deepbelief.DeepBelief.JPCNNLibrary;

This class contains a list of Java functions that correspond to exactly to the C interface functions. The class code is available in the AndroidLibrary folder, and you should be able to rebuild it yourself by running ant, but here are the definitions using JNA types:

Pointer jpcnn_create_network(String filename);
void jpcnn_destroy_network(Pointer networkHandle);
Pointer jpcnn_create_image_buffer_from_file(String filename);
void jpcnn_destroy_image_buffer(Pointer imageHandle);
Pointer jpcnn_create_image_buffer_from_uint8_data(byte[] pixelData, int width, int height, int channels, int rowBytes, int reverseOrder, int doRotate);
void jpcnn_classify_image(Pointer networkHandle, Pointer inputHandle, int doMultiSample, int layerOffset, PointerByReference outPredictionsValues, IntByReference outPredictionsLength, PointerByReference outPredictionsNames, IntByReference outPredictionsNamesLength);
void jpcnn_print_network(Pointer networkHandle);

Pointer jpcnn_create_trainer();
void jpcnn_destroy_trainer(Pointer trainerHandle);
void jpcnn_train(Pointer trainerHandle, float expectedLabel, float[] predictions, int predictionsLength);
Pointer jpcnn_create_predictor_from_trainer(Pointer trainerHandle);
void jpcnn_destroy_predictor(Pointer predictorHandle);
int jpcnn_save_predictor(String filename, Pointer predictorHandle);
Pointer jpcnn_load_predictor(String filename);
float jpcnn_predict(Pointer predictorHandle, Pointer predictions, int predictionsLength);

There are a few quirks to using the interface that the example code demonstrates how to work around. jpcnn_create_network() requires a standard filename path, but to distribute the network with an application it needs to be an asset, and because that may be compressed and part of an archive, there's no way to get a path to it. To fix that, initDeepBelief() copys the file to the application's data directory:

AssetManager am = ctx.getAssets();
String baseFileName = "jetpac.ntwk";
String dataDir = ctx.getFilesDir().getAbsolutePath();
String networkFile = dataDir + "/" + baseFileName;
copyAsset(am, baseFileName, networkFile);
networkHandle = JPCNNLibrary.INSTANCE.jpcnn_create_network(networkFile);

This has some overhead obviously, so one optimization might be to check for the existence of the file and only copy it over if it doesn't already exist.

jpcnn_create_image_buffer_from_uint8_data() needs a plain byte array, and the classifyBitmap() function shows how you can extract what you need from a normal Bitmap object:

final int width = bitmap.getWidth();
final int height = bitmap.getHeight();
final int pixelCount = (width * height);
final int bytesPerPixel = 4;
final int byteCount = (pixelCount * bytesPerPixel);
ByteBuffer buffer = ByteBuffer.allocate(byteCount);
bitmap.copyPixelsToBuffer(buffer);
byte[] pixels = buffer.array();
Pointer imageHandle = JPCNNLibrary