END-USER/EXPERT BOT SOLUTION

The end user/expert bot solution is a 2 bot product that allows the end users to query a knowledge base using natural language. If no answer is found on the Knowledge base or if the user thinks that the answer is wrong, a team of experts is contacted using the second bot of the solution.

The expert bot allows the team of experts to answer to the end users' questions and to update and improve the knowledge base. All this interaction is also done through the usage of natural language.

This soultion allows for several end user and expert channels, however currently only has been tested for the following:

• End User:

  • Skype
  • Web Chat

• Expert:

  • Microsoft Teams

ARCHITECTURE AND DATA MODEL

The full solution comprises of the following items:

• End User Bot
• Expert Bot
• Knowledge Base
• Table storage
• Natural Language Understanding service
• Queue
• Event Hub
• Stream Analytics

Tables

There are several No-SQL tables supporting the solution. Their main purpose is:

• Storing user data;
• Store the mappings between users and the questions they made;
• Store the questions that need be answered by the experts;
• Store the End user feedback regarding the response from the Knowledge base;
• Store Auditing logs for further processing by BI tools
• Store the bot context and state.

Users

The users table stores information about the End Users of the system, namely the user id, the channel, the last time the user was active and if the user has the notifications turned on.

The Users table is partitioned by ChannelId and each row is identified by the UserId. (The pair ChannelId/UserId is always unique)

When the End User bot receives a new message, he either creates or updates a record for the user, setting the last active time to the time the message was received. When creating a user, the notifications are always enabled.

Still Interested Table

The still interested table is used to store the mapping between the users and the questions they make. This table contains the questions, the user, the channel and whether the answer was already given or not. (This flag is used to decide if the message coming from the experts is an update to the answer or the first time the answer is given).

The table is partitioned by the Question scentence and each row is identified by {channelId:userId}.

Whenever a new message arrives from the "Experts to User" queue, the bot retrieves all the users that are interested on the question (retrieves that question's partition).

Unanswered Questions Table

The unanswered questions table is used to store which questions are pending an answer from the Experts. The table contains the question, the answer that was given from the Knowledge Base (if it exists), the question that matched on the Knowledge base (if it exists) and the date the question was posted to the experts. There is no need to store the user information since it will be stored on the still interested table.

The table is partitioned using the hash of the question and each row identifier is calculated as the hash of the question that matched on the knowledge base. This ensures that there are no duplicates of the same question.

Whenever a new message arrives to the Expert Bot, a new record for that question is inserted on the table.

Expert Bot Channels

The expert bot channels table stores a list of all the channels that are available for the Expert bot. This allows the bot to broadcast a question from a user to a multitude of Expert teams and individual contributers (through inividual channels such as skype).

This table is partitioned by the bot framework channelId and contains for each channel a unique identifier (guid) and the JSON serialization of a bot framework's ConversationReference object.

Everytime a message is sent to the Expert Bot, the bot checks if the channel is already registered. If it's a new channel, a new record is created

Feedback table

The feedback table holds the information about the user feedback for a given pair question/answer. This feedback can be either negative (no answer was found or the answer was deemed wrong) or positive (the answer given by the Knowledge base was correct).

The table is partition by Positive and Negative feedbacks, and the key inside each partition is given by a unique id (GUID) that is generated for every question. The table also stores the answer given by the knowledge base, if it existed.

Auditing

The auditing tables are used to store data that then will be used to produce reports of the usage of the solution.

There are 2 auditing tables:

• ExpertAuditing - stores all the auditing information for the expert bot;
• EndUserAuditing - stores all the auditing information for the end user bot;

Bot Data

The bot data tables are the tables that are used by the bot framework to mainitain the Bot context and state.

There are bot data tables on this solution, one for each bot. Each one of them is a no-SQL table on Azure Table Storage. For more details on these tables check this link.

The tables are:

• enduserbotdata - stores the bot data for the End User bot;
• expertbotdata - stores the bot data for the Expert bot;

Queues

This solution uses Azure Storage queues to ensure both bots can exchange messages without being tightly coupled. Since the messages are small in size and there are no needs of storing them for a long time the Service Bus Queue was not taken into account.

There are 3 queues in total in this solution:

• User to Expert queue
• Expert to User queue
• Notifications queue

User to Expert Queue

The user to expert queue is implemented as an Azure Storage Queue.

The object that is sent in the queue has the following fields:

• Question - the question of the user;
• OriginalAnswer - the answer that was given by the Knowledge Base (if exists);
• OriginalQuestion - the question from the Knowledge Base that matched the user's question (if exists).
• ExpertAnswer - the answer of the expert (it will be blank);
• Timestamp - when the question was posted;
• MessageType - If the message is about a Wrong Answer from the Knowledge Base or if is about a question that has no answer.

Expert to User Queue

The expert to user queue is also implemented as an Azure Storage Queue. The object that is sent in the queue has the same structure as the object sent on the User to Expert Queue. The ExpertAnswer field will be filled this time.

Notifications queue

The notifications queue is also implemented as an Azure Storage Queue. The object that is sent in the queue only has one field:

• TextToSend

Blobs

There are 2 blob containers on the solution:

• Configurations
• Files

These containers were implmented on Azure Blob Storage

Configurations

The configurations container contains the configuration files for both bots.

As stated before, for each bot, there are 2 files - the configuration file and the message file. Hence the configurations container has the following contents:

• End User Bot Configuration;
• End User Bot Messages;
• Expert Bot Configuration;
• Expert Bot Messages;

Files

The files container is where the files that can be downloaded by the users are stored. Currently there will only be 2 files on this container:

• Private Knowledge Base file
• Public Knowledge Base file

KNOWLEDGE BASE

The Knowledge Base is the core element of the solution. The knowledge base is a set of question/answer pairs with what both bots interact.

Currently the knowledge base is implemented using the Microsoft QnA Maker. This product has a Question/Answer list and has a small component of NLU, that ensures that an exact match with the question on the knowledge base is needed.

EVENT HUBS

The event hubs are a tool that allows the processment of several events at the same time. On this solution, the Event Hub is used to send auditing events from both bots.

To ensure that the data is logically separated, there are 2 event hubs, one for the EndUser events and other for the Expert events.

STREAM ANALYTICS

The Stream Analytics component is responsible for reading the events from the event hub and place them in the respective table (ExpertAuditing or EndUserAuditing).

To do so, the stream analytics job gets the data from each one of the event hubs and place it on the respective Azure table.

END USER BOT

The end user bot is a bot designed to work on several channels, namely Skype and Web Chat, that allows the end user to query a knowledge base for information. If the user doesn't find the information that he uas looking for, or if the information is not correct, he has the ability to notify a team of experts to insert the correct answer on the knowledge base. The bot uses an NLU engine to determine what are the intents of the end user. Currently, the bot has available the following intents:

• AskQuestion -- The user wants to query the knowledge base and notify the experts team if the answer is missing or is wrongly given;
• DownloadKnowledgeBase -- The user wants a file with all the knowledge base. In the end user bot the file only contains the list of questions stored on the knowledge base. The file is in a tsv (tab separated values) format;
• TurnOnNotifications -- The user wants to receive notifications about updates on the answer of his questions.
• TurnOffNotifications -- The user wants to stop receiving notifications about updates on the answer of his questions.

EVENT FLOWS

AskQuestion flow of events

The AskQuestion flow of events is the following:

1. User writes on the Skype window (eg. What is Azure Stack?);
2. The bot understands that the intent is AskQuestion and fowards the query to the Knowledge Base;
3. If the knowledge base didn't find the answer, a message is automatically posted to the Experts