Optimist Prime Facebook Messenger Bot

Optimist Prime is a Facebook Messenger Bot that supports Voice Recognition, Natural Language Processing and features such as: search nearby restaurants, search trending news, transcribe and save memos to the cloud. It also save user data (with permissions, of course) such as favorite locations, can provide customized greetings (acknowledging user's time in any time zone, i.e. Good morning/Good evening) and entertaining responses, etc.

(For a simpler "echo bot" proof-of-concept implementation of the Facebook Messenger Bot, check out this simplified project with a 10-minute tutorial)

Table of Contents

• Features
• Screenshots
• Usage
  • Dependencies, Database and API keys
  • Deploying to the cloud
  • Voice Recognition
  • Natural Language Processing
  • Custom features
    • 1. Business/Restaurant Search
    • 2. Trending News Search
    • 3. Memo
• Appendix
  • Facebook Messenger API
  • Sample Facebook Messenger API Messages
    • 1. Text
    • 2. Audio
    • 3. Location
  • Discussion
    • The nitty-gritty detail of implementing voice recognition & scalability

Features:

• Voice Recognition
• Understanding commands with Natural Language Processing and contextual follow-up
• Business/Restaurant Search
• Trending News Search
• Speech-to-Text note taking (with Cloud access)
• Conversational chit-chat

Screenshots:

Demo (click on Message to start chatting with it)

Usage

Note: Optimist Prime is implemented with different APIs for features like user management, voice recognition, restaurant search, trending news search, so it takes some time to config & get it up and running. For a more basic "echo bot" that responses to you whatever you say to it, use facebook-echobot.py, or head over to Facebook's own Messenger app Quick Start. The echo bot is useful to get a quick glance of the fundamental ideas behind a Facebook Messenger Bot.

Dependencies, Database and API keys

In order to build your own bot with all features of Optimist Prime, you'll need a few set-ups:

0. Install dependencies: pip install -r requirements.txt (preferably getting into your virtual environment virtualenv/venv - read all about pip and venv here)
1. Create a Facebook Page: A bare-bone Page to "assign" the Bot to will do. The Bot will actually be this page, i.e. you'll be "talking" to the page
2. Create a Facebook App, get its Page Access Token (details at Facebook's Quick Start)
3. Create a MongoDB database (User management, Conversational Context management, Logging), a local MongoDB is fine (Tutorial to set up a local instance). I used Heroku's mLab MongoDB. It'll take you 10 minutes to get a Heroku account and set up a MongoDB database there.

Then a few configurations in config.py:

1. MongoDB database credentials (created above)
2. Yelp API key (Business/Restaurant Search feature): Get one here (More details below, as Yelp now has a stable v2 API and a developer preview v3)
3. IBM Watson Speech-to-Text API username & password: Get one here (More details below)
4. Simsimi: Get one here (Free 7-day trial key)
5. Define your own local config: create a folder called instance, and create another config.py file in it. (More on Flask configurations)

To run locally, as simple as:

python facebookbot.py 3000

Or with gunicorn (as I do on Heroku) (Flask and gunicorn tutorial)

gunicorn facebookbot:app -b localhost:3000

Now that you've got the bot running, you'll need to set up a webhook for Facebook to send messages the bot receives to. This could be done with ngrok, which will set up a secure tunnel to your localhost server:

./ngrok http 3000

Get the https URL (Facebook requires https secured webhooks) and subscribe your Facebook App to this webhook. The verification token is your own token defined in OWN_VERIFICATION_TOKEN in config.py.

Deploying to the cloud

I've provided the Procfile for deployment on Heroku. You can create a Heroku app, spin up a free dyno and deploy your own Optimist Prime with this tutorial.

For the voice recognition to work, we'll need to include ffmpeg on our Heroku dyno, which could be done by adding a Heroku Buildpack to your app's Settings tab on Dashboard: https://github.com/jonathanong/heroku-buildpack-ffmpeg-latest.git

Finally, set your environment variable for the path to ffmpeg:

heroku config:set FFMPEG_PATH=/app/vendor/ffmpeg/ffmpeg

Or on your app’s settings tab on Dashboard:

Now you're ready to deploy. Tutorial on how to deploy with Heroku and git.

In later iterations, all you need to do with Heroku is the glorious 3 lines:

git add .
git commit -am "Awesome commit"
git push heroku master

Amazon Web Service: I'm a fan of AWS and have had great experience with Beanstalk. However, if you want to use AWS, you'll need to go the extra mile of obtaining an SSL cert to have a secured webhook. For the purposes of Optimist Prime, I decided to go with Heroku instead, since it readily provides a https connections.

Voice Recognition

The Voice Recognition is implemented with both IBM Watson's Speech-to-Text API and Google Cloud Speech API (default to IBM Watson as Google Cloud Speech is still in Beta, and my tests showed Watson so be more accurate). The current implementation is based on their RESTful methods (both support real time processing with WebSocket and WebRTC, respectively). Both are available for free at development-level use.

To use IBM Watson's Speech-to-Text, you'll need to create a IBM Bluemix account and add the service to your account, then retrieve the API's username and password. Lastly, copy these credentials to Speech/credentials.py.

To use Google Cloud Speech API, the process is a little bit more complicated as you'll need to export Google's credentials as a environment variable. However, the whole process is well-documented by Google over here. As soon as you have got the Service Account key file (json) and exported GOOGLE_APPLICATION_CREDENTIALS environment variable to the key file's location, you're set to go.

To switch between IBM Watson and Google for speech recognition: Setting the environment variable as follow:

export FB_BOT_STT_API_PROVIDER=GOOGLE
export FB_BOT_STT_API_PROVIDER=IBM

The result text processed by this Speech-to-Text API is then returned just like a text message the bot receives, which then goes through NLP for detecting commands/conversations.

Natural Language Processing

Optimist Bot receives commands from users as both text and voice input, and understands commands in natural language.

This is done by using the pattern NLP library, which allows the bot to deconstruct the user's text input and recognize parts of speech. For now, the model for categorizing commands are simple with stopwords and sentence structures, but as our data grows, we can start building more complex machine learning categorization for each function.

The command system allows users to use the following features, all of which are under the Utils folder.

Custom features

1. Business/Restaurant Search

Example commands:

		irish bar near by.
		find chinese restaurants.
		find me a good coffee shop around here.
		show me Chinese food close by.
		find mexican restaurants near here.
		I want to have vietnamese food tonight.
		is there a korean bbq nearby?
		what are some cambodian grill close by?
		find an ethiopian restaurant.
		I want mediterranean food.
		find a Target.
		find me a KFC around.
		I'd like to eat at McDonalds.
		find me some fast food places in ohio city.
		find me a brewery near downtown san francisco.

After receiving the command, Optimist Prime would ask for your location. You can input either a text/voice-based location name or send