Pupillometry in the web browser

This JavaScript library detects, tracks and measures the radius of the 2 pupils of the eyes of the user. It uses a standard 4K webcam and it runs into the web browser, fully client side. It fully uses the GPU hardware acceleration thanks to WebGL, for each computation step:

• for head detection (using a deep neural network),
• for iris segmentation (using Circle Hough Transform),
• for Pupil segmentation (using a custom ray tracing algorithm)
• for debug rendering

This library is lightweight: it captures the image of the webcam and gives the relative radiuses of the 2 pupils as output (ie (radius of the pupil)/(radius of the iris). It does not include a measurement recorder or a chart plotter. We want to keep it framework agnostic so it is integrable into all environments. We have included a light plotter and a light recorder among the helper scripts in the helpers path for the demonstrations.

Table of contents

• Pupillometry
• Features
• Architecture
• Demonstrations
• Sliders usage
• Hardware
  • The camera
  • The lighting
  • The experimental setup
• Experimental results
  • Light intensity
  • Calm or fear
• Specifications
  • Get started
  • Optionnal init arguments
  • Error codes
  • The returned objects
  • Miscellaneous methods
• Hosting
• About the tech
  • Under the hood
  • Compatibility
• License
• References

Pupillometry

The pupillometry is the measurement of the radius of the pupil of the eyes. It can depend on several factors:

• the ambient luminosity,
• the state of consciousness (the pupil can dilate with some drugs),
• the mental state (attractivity, liar detection, ...).

This JavaScript library measures the pupillometry in the browser. But it should be used in a rigorous experimental setup: you should use a test sample and a validation sample, you should control the luminosity of the place and you should apply all the rules which makes an experiment a scientific experiment. This measure is not directly usable, and it is always noisy. You should read related scientific publications before implementing any experimental setup.

Features

Here are the main features of the library:

• face detection,
• face tracking,
• face rotation detection,
• very robust for all lighting conditions,
• video acquisition,
• measurement of the radiuses of the pupils,
• debug view.

Hardware

We strongly advise to use this library with a desktop or a laptop computer with a dedicated graphic card. Indeed, the video processing is quite heavy and most mobile devices won't be powerful enough. Moreover, a mobile device may move too much and the tracking may be lost from time to time. For our tests, we use a 2015 laptop with a Nvidia GTX960M and Chrome 67 and it works like a charm.

Then we need a very high resolution camera, typically an 4K camera. Indeed, should be able to measure the pupil with a significant amount of pixels. Videoconferencing 4K cameras are not fitted for this purpose because their field of view is too large. We use a 4K CCTV camera with a custom adjustable lens to reduce the field of view. This solution is cheaper, and most CCTV cameras have a big advantage in our case: they does not have a filter to block the infrared and the pupil is more separable from the iris in the infrared field.

We propose a setup until all laptops are equipped with 4K infrared cameras :). You can buy all the parts for about $100 and we expect a decrease in the price of 4K cameras. It is far cheaper than specialized pupillometry devices (some are over $20000). If you succeed in running this library with a lighter setup, we are very interested to add the information here!

You will need small screwdrivers and a bit of common sense to assemble everything. It is easier than building some Ikea furnitures.

The camera

• The 4K (= 8MP) camera module is a ELP-USB8MP02G-L75. ELP make great and cheap camera modules, very compliant with Linux drivers. The specifications are here and you can buy it on Amazon here. You can plug this module directly on the USB port,

• A lens for the camera, otherwise the field of view is too large (75 degrees). You can buy it on Amazon here. Here are its specifications:

  • Len Type: Manual Iris Lens
  • Focal Length: 9 ~ 22mm
  • Imager Size: 1/3"
  • Mount Type: M12 x 0.5
  • Aperture Range: F / 1.4
  • M. O. D: 0.2m

• A metal case to protect the camera module. you can buy it on Aliexpress here

• A support a bit heavy like a SmallRig Switching Plate Camera Easy Plate measuring SmallRig Switching Plate Camera Easy Plate: buy it here on Amazon and do not forget the screws ! The screws are here on Amazon

Now, mount all the stuffs to get a powerful 4K infrared enabled camera:

• take the camera module and remove the screws fixing the plastic lens support. After that, the CMOS captor should be naked on the electronic board like in this picture. Note: you can also directly buy the ELP camera module without the lens (ref: ELP-USB8MP02G). Be very careful: DO NOT PUT YOUR FINGERS ON THE CMOS CAPTOR ! And do the operation in a dust free environment.

• put the camera module into the metal case and close it using the provided screws,

• screw the lens to the camera module. Screw the metal case handle. Be careful: there are short and long screws, the long are used to fix the handle.

• fix the metal case handle to the cheese plate rig. The central hole is too small for the small rig screws. You can either enlarge it if you have the appropriate tool, or fixing the handle by putting 2 screws on each side to fix it.

You can found a picture of the mounted camera here.

The lighting

The light should be red because the pupils do not contract on red light. A light from below is great because it won't make reflections on the pupil like a direct one, and the eyelids will not make shadows.

• a clip light with a reflector is perfect (Bulb E26, up to 60 Watts, with a 8.5 inch aluminium reflector ). You can buy it on Amazon here,

• a red bulb like this one.

The experimental setup

The total cost of all this hardware (camera + 1 light) is (59.99+14.48+6+9.99+5.99)+(8.99+7.04) = $112.48 and most of the hardware is reusable for other projects.

The ideal position for the camera is centered horizontally, and below the head, looking up. Place it at the bottom of the screen if it is possible. Avoid to put the camera too high: it will then look down and the eyelids will mask the eyes. On contrary, if the camera is looking up, it would not be bothered by eyelids.

Avoid any front lighting: it makes reflections on the pupil and it disturbs the pupil radius detection algorithm. The best place is on a desktop or a table facing a wall without windows or lights on it.

Fix the lights on the deskop using the clips, one a each side of the user. Then launch the demonstration provided in this repository. You start in the Debug view. This view is important for calibration and it shows the 3 levels of detection:

• The global face detection using a neural network,
• The iris segmentation using circle hough transform,
• The pupil radius measurement using a custom ray tracing algorithm.

Do the following operations to ajust consecutively the 3 levels of detection:

• Check that the camera used is the 4K camera and not the small laptop camera,
• Put your head in front of the computer, not too close to the camera,
• Adjust the lens (2 settings: zoom and focus) in order to have an image not blurry (the edge border detection in red can help you to find a good setting),
• The face image should be centered and your face should be detected,
• Adjust the 3 first sliders about the Iris detection and segmentation. At the end the Iris should be detected and cropped correctly. The result should be stabilized,
• Adjust the lighting (the red lights position) so that the pupil should be black, without too strong reflections, and the iris should be sufficiently illuminated to be separable from the pupil. You can also play with the camera settings (with Linux/Debian, launch v4l2ucp /dev/video1 and replace automatic exposure with manual one. This a screenshot of our settings),
• Adjust the last slider, the Pupil detect sensitivity. The detection of the pupil is shown by a lime circle on the iris view.

The detection works better without wearing glasses. If you still have to wear it, they should be very clean in order to not disturb the pupil detection.

Architecture

• /demos/: source code of demonstrations,
• /dist/: heart of the library:
  • jeelizPupillometry.js: main minifi