JavaScript/WebGL lightweight and robust face tracking library designed for augmented reality face filters

This JavaScript library detects and tracks the face in real time from the camera video feed captured with WebRTC. Then it is possible to overlay 3D content for augmented reality applications. We provide various demonstrations using the main WebGL 3D engines. We have included in this repository the release versions of the 3D engines to work with a determined version (they are in /libs/<name of the engine>/).

This library is lightweight and it does not include any 3D engine or third party library. We want to keep it framework agnostic so the outputs of the library are raw: if the face is detected or not, the position and the scale of the detected face and the rotation Euler angles. But thanks to the featured helpers, examples and boilerplates, you can quickly deal with a higher level context (for motion head tracking, for face filter or face replacement...). We continuously add new demonstrations, so stay tuned!

<!-- If you need a custom development service using this library, you can submit the [FaceFilter development request form](https://forms.gle/kktPyojpJbwSSPED7). We will get back to you quickly. -->

Table of contents

• Features
• Architecture
• Demonstrations and apps
  • Included in this repository
  • Third party
• Specifications
  • Get started
  • Optional init arguments
  • Error codes
  • The returned objects
  • Miscellaneous methods
  • Multiple faces
  • Multiple videos
  • Optimization
  • Changing the 3D engine
  • Changing the neural network
  • Using module
• Integration
  • With a bundler
  • With JavaScript frontend frameworks
    • With REACT and THREE Fiber
    • See also
  • Native
• Hosting
  • The development server
  • Hosting optimization
• About the tech
  • Under the hood
  • Compatibility
• Articles and tutorials

<!-- * [Developer support plans](#developer-support-plans) --> <!-- * [Jeeliz Partner Network](#jeeliz-partner-network) -->

• License
• References

<p align="center"> <img src='https://user-images.githubusercontent.com/11960872/37533324-cfa3e516-2941-11e8-99a9-96a1e20c80a3.jpg' /> </p>

Features

Here are the main features of the library:

• face detection,
• face tracking,
• face rotation detection,
• mouth opening detection,
• multiple faces detection and tracking,
• very robust for all lighting conditions,
• video acquisition with HD video ability,
• mobile friendly,
• interfaced with 3D engines like THREE.JS, BABYLON.JS, A-FRAME,
• interfaced with more accessible APIs like CANVAS, CSS3D.

Architecture

• /demos/: source code of the demonstrations, sorted by 2D/3D engine used,
• /dist/: core scripts of the library:
  • jeelizFaceFilter.js: main minified script,
  • jeelizFaceFilter.module.js: main minified script for use as a module (with import or require),
• /neuralNets: trained neural network models:
  • NN_DEFAULT.json: file storing the neural network parameters, loaded by the main script,
  • NN_<xxx>.json: alternative neural network models,
• /helpers/: scripts which can help you to use this library in some specific use cases,
• /libs/: 3rd party libraries and 3D engines used in the demos,
• /reactThreeFiberDemo/: NPM/React/Vite/Three-Fiber boilerplate.

Demonstrations and apps

Included in this repository

These demonstration are included in this repository. So they are released under the FaceFilter licence. You will probably find among them the perfect starting point to build your own face based augmented reality application:

<!-- These demos are NOT maintained anymore: --> <!-- * Daft Punk (put the iconic helmet): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/daft_punk/) * Star Wars: Darth Vader: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/star_wars/) * Harry Potter (say "Lumos!"): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/harry_potter/) * Halloween Spiders (you've got a spider in your mouth): [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/halloween_spider/), [source code](/demos/threejs/halloween_spiders) -->

• REACT/THREE FIBER boilerplate: /reactThreeFiberDemo

• BABYLON.JS based demos:

  • Boilerplate (displays a cube on the user's head): live demo, source code

• THREE.JS based demos - specific README about THREE.js based demo problems:

  • Boilerplates:

    • Boilerplate (displays a cube on the user's head): live demo, source code
    • Boilerplate with only 1 <canvas> element: live demo, source code
    • Same boilerplate but using neuralNets/NN_4EXPR_1.json as neural net, and displays 4 expressions: live demo, source code
    • Multiple face tracking: live demo, source code
    • GLTF fullscreen demo with HD video: live demo, source code

  • AR 3D demos:

    • Werewolf (turn yourself into a werewolf): live demo, source code
    • Angel/Demon (discover who of the angel or demon will win in this animated scene): live demo, source code
    • Anonymous mask and video effect: live demo, source code
    • Rupy Motorcycle Helmet VTO: live demo, source code
    • Dog: live demo, source code
    • Butterflies animation: live demo, source code
    • Clouds above the head: live demo, source code
    • Casa-de-Papel mask: live demo, source code
    • Miel Pops glasses and bees: live demo, source code
    • Football makeup: live demo, source code
    • Tiger face filter with mouth opening detection (strong WTF effect): live demo, source code
    • Fireworks - particules: live demo, source code

  • face painting or deformation:

    • Face deformation: live demo, source code
    • Face cel shading: live demo, source code

  • demos linked with tutorials:

  <!-- * Luffy's Hat: [live demo](https://jeeliz.com/demos/faceFilter/demos/threejs/luffys_hat_part2/), [source code part 1](/demos/threejs/luffys_hat_part1/), [tutorial part 1](https://jeeliz.com/blog/creating-a-snapchat-like-filter-with-jeelizs-facefilter-api-part-1-creating-your-first-filter/), [source code part 2](/demos/threejs/luffys_hat_part2/), [tutorial part 2](https://jeeliz.com/blog/creating-a-snapchat-like-filter-with-jeelizs-facefilter-api-part-2-user-interactions-and-particles/) -->

  • Statue Of Liberty: live demo, source code, interactive tutorial

  • Matrix: live demo, source code, tutorial in French<!--, [tutorial in English](https://jeeliz.com/blog/tutorial-javascript-webgl-webcam-facial-filter-on-the-theme-of-matrix/) -->

  • misc:

    • Head controlled navigation: live demo, source code
    • Glasses virtual try-on: live demo, source code

• A-FRAME based demos:

  • Boilerplate (displays a cube on the user's head): live demo, source code

• CSS3D based demos:

  • Boilerplate (displays a <DIV> element on the user's head): [