Annotator

Annotator is a back and front end solution for performing annotation of images with discrete objects (e.g. points, lines, circles, polygons, polylines, bounding boxes, etc.). A data abstraction for discrete annotations based on point sets is stored using MongoDB. The data and the application is served using Node.js, ExpressJS and Mongoose. The front end application is a single-page application built using HTML/CSS, Javascript, and SVG. Specifically it uses Backbone.js for synchronization with the database and interface control while visualizing the actual annotations using SVG and D3.js. Parts of the interface also utilize jQuery and jQuery UI.

Installation

The following instructions were tested on Ubuntu 12.04 LTS.

Install Node.js

1. Go to http://nodejs.org/, download the install file
2. Unpack with tar zxvf node-v0.10.31.tar.gz
3. cd node-v0.10.31
4. ./configure
5. make
6. sudo make install

• You might get a complaint while installing the man page: OSError: [Errno 20] Not a directory: '/usr/local/share/man/man1/node.1'. I simply did sudo mv /usr/local/share/man/man1 /usr/local/share/man/man1.backup.

Install MongoDB

Follow the instructions for installing on Ubuntu, reproduced below.

1. sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
2. echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
3. sudo apt-get update
4. sudo apt-get install mongodb-org
5. sudo service mongod start
6. Wait a bit and then verify that the installation succeeded and the serviced started by running mongo. That will take you into a Mongo command-line interface where you can run show dbs which should return

admin  (empty)
local  0.078GB

Install Annotator

1. Clone the Annotator repository: git clone https://github.com/vietjtnguyen/annotator.git
2. cd annotator
3. Install Node dependencies. npm install
4. If you are serving this from a non-root URL then you need to edit the first code line of ./public/app/js/annotator.js. Normally the application assumes that you are hosting it at the root of your domain (e.g. http://my.domain.com/) but if you are not (e.g. http://my.domain.com/annotator/) then you need to specify rootUrl (e.g. var rootUrl = "annotator";).
5. Start the server by running npm start in the repository's root. By default the application binds to port 3000. If you need to bind it to another port then specify the PORT environment variable (e.g. export PORT=80 in bash).

Install Dataset

1. Make sure ImageMagick is installed (needed for the convert command): sudo apt-get install imagemagick
2. Place dataset (folder with images) in the ./dataset folder. This isn't necessary but is recommended. Let's assume your folder is called myDataset (i.e. it is at ./dataset/myDataset).
3. Run ./install_dataset.bash public/image ./dataset/myDataset. This script will find every bmp, gif, jpg, jpeg, png, tif, and tiff in your dataset folder. It then uses convert to get the width and height and shasum to get the SHA1 hash of the image file. This information including image name, file, and URL are saved as entries in the local Mongo database in the annotator database in the images collection. This save is done using mongoimport. Finally it creates a symlink in ./public/image to the original image file. Images are served statically from ./public/image and are named based on their SHA1 hash.

Note that image names must be unique, but SHA1 hashes do not have to be unique (so the same image file can have different names).

Keep in mind that there is a 16 MB limit to mongoimport when using --jsonArray which the install_dataset.bash script uses. For reference, the JSON file for all PASCAL 2010 images (about 20k images) is 4.6 MB.

An example dataset installation script is provided in ./dataset/install_york_dataset.bash.

Create Image Set

Image sets are ordered lists of image IDs (as in MongoDB IDs). The create_image_set.js script can be used to create image sets from lists of Mongoose queries. This allows for the creation of lists from lists of image names, image SHA1 hashes, filenames, etc.

A Mongoose query is just a Javascript hash object of desired field-value matches. For example, query {"name": "P1020171"} returns an image that has field name equal to P1020171. Mongoose supports more complex queries, but in creating an image set we expect each query to return only one image. If the query returns empty or error the script retries two more times (in case there were locking issues) before skipping the image and leaving it out of the final image set.

The script takes in a JSON file which contains an array of Mongoose queries to find one image in the database. The order of queries is preserved in the final image set.

The second argument is the name to assign the image set in the database. This name is used to reference the image set and thus must be URL friendly (though this is not enforced). For example, an image set named YorkUrbanDB can be accessed at URL /parallel-lines/set/YorkUrbanDB/1.

./createImageSet.js jsonArrayFile.json imageSetName

The following is an example image set JSON file:

[
{"sha": "98bb60c011a38c76f71f409a58fa5bf5cf70e115", "name": "P1020171", "filename": "datasets/YorkUrbanDB/P1020171/P1020171.jpg", "url": "/image/98/bb60c011a38c76f71f409a58fa5bf5cf70e115.jpg", "width": 640, "height": 480, "comment": ""}, 
{"sha": "4989e260e387455d66632c1327a46d56930f1e74", "name": "P1020177", "filename": "datasets/YorkUrbanDB/P1020177/P1020177.jpg", "url": "/image/49/89e260e387455d66632c1327a46d56930f1e74.jpg", "width": 640, "height": 480, "comment": ""}, 
{"sha": "53fe4189ac3265027a914b5a2c5cd03b4f90c4d0"}, 
{"sha": "eb0dafbcdba94b011d7ce6af3b71827748abe2d9"}, 
{"name": "P1020819"}, 
{"name": "P1020822"},
...
{"name": "P1080119"}
]

The JSON file produced by install_dataset.bash is valid input to create_image_set.js.

Note that the input file must be valid JSON. Thus the trailing comma in the array cannot be present. The underscore-cli utility can be handle for making sure the JSON outputs correctly (see ./datasets/pascal_list_to_image_set.bash).

Getting Data

You can extract the data saved by visiting a special endpoint: /parallel-lines/report.json. This will export all of the saved annotation data as a JSON file. The JSON object is keyed by image name. Each image is an array of objects with a points, group, and description attribute. See the example below.

{
  "P1020177": [
    {
      "points": [ { "x": 231.5 "y": 106.5 }, { "x": 325.5 "y": 214.5 } ],
      "group": "53fe890c33c7de9713b2948c",
      "description": ""
    },
    {
      "points": [ { "x": 249.5 "y": 231.5 } { "x": 337.5 "y": 131.5 } ],
      "group": "53fe890c33c7de9713b2948c",
      "description": ""
    }
  ],
  "P1080015": ...
}

The group is just a randomly generated hexadecimal GUID to ensure uniqueness when groups are created. The group name has no semantic meaning.

About

Computer vision and machine learning research is dependent (depending on methodology) on labeled datasets as a ground truth for both training and testing.

Early datasets were image level annotations where an entire image was labeled with a class (e.g. MNIST Digits, Caltech 101). Scene classification datasets can also be crudely considered image level annotations. Object detection algorithms require region level annotations, usually in the form of a bounding box around objects in an image. Object segmentation takes it further by labeling object silhouettes with arbitrary shaped regions. These arbitrary shaped regions either take the form of a set of polygons (with no convexity constraint) or pixel level labels, though it can be argued that these formats are interchangeable.

Labeling these datasets is a labor intensive task with labeling tools often rewritten from scratch. Many of the tools I have seen written for annotation tasks are one off projects written using MATLAB's GUI. The goal of this project is to develop a fairly flexible annotation system that is web based and allows for different annotation types. Having a web application as the annotation tool allows us to leverage a huge ecosystem of web technologies. It also lets labelers access the tool in a distributed manner without the need for MATLAB.

Concepts

We abstract single points, lines, polylines, polygons, and bounding boxes into just point sets. Point sets are ordered arrays of two dimensional points with x and y attributes. A point set is an abstract notion. Derived models such as lines enforce constraints such as having a maximum of two points. Bounding boxes can be represented with two points as well.

Developer Notes

All of the below is not rigorous or final.

Commands

• Add point set
• Remove point set
• Add group
• Remove group
• Add point to point set
• Remove point from point set
• Change point set group membership

API

Generic end points:

• /{annotation_name}/{image_id}
• /{annotation_name}/name/{image_name}
• /{annotation_name}/set
• /{annotation_name}/set/{set_name}
• /{annotation_name}/set/{set_name}/{image_index}
• /api
• /api/image
• /api/image/{image_id}
• /api/image/name/{image_name}
• /api/{annotation_name}
• /api/{annotation_name}/set
• `/api/{an