ThePhish

<div> <p align="center"> <img src="pictures/logo.png" width="800"> </p> </div>

ThePhish is an automated phishing email analysis tool based on TheHive, Cortex and MISP. It is a web application written in Python 3 and based on Flask that automates the entire analysis process starting from the extraction of the observables from the header and the body of an email to the elaboration of a verdict which is final in most cases. In addition, it allows the analyst to intervene in the analysis process and obtain further details on the email being analyzed if necessary. In order to interact with TheHive and Cortex, it uses TheHive4py and Cortex4py, which are the Python API clients that allow using the REST APIs made available by TheHive and Cortex respectively.

Table of contents

• Overview
• ThePhish example usage
  • A user sends an email to ThePhish
  • The analyst analyzes the email
• Implementation
• Installation
  • Install it using Docker and Docker Compose
  • Install it from scratch
• Configure the analyzers
  • Configure the levels of the analyzers
  • Tested analyzers
  • Enable the MISP analyzer
  • Enable the Yara analyzer
• Enable the Mailer responder
• Use the whitelist
• Contribution to TheHive4py
• Contribution to Cortex-Analyzers
• License
• Academic publications
• Who talks about ThePhish
• GitHub repositories mentioning ThePhish
• Credits

Overview

The following diagram shows how ThePhish works at high-level:

<img src="pictures/overview.png" width="700">

1. An attacker starts a phishing campaign and sends a phishing email to a user.
2. A user who receives such an email can send that email as an attachment to the mailbox used by ThePhish.
3. The analyst interacts with ThePhish and selects the email to analyze.
4. ThePhish extracts all the observables from the email and creates a case on TheHive. The observables are analyzed thanks to Cortex and its analyzers.
5. ThePhish calculates a verdict based on the verdicts of the analyzers.
6. If the verdict is final, the case is closed and the user is notified. In addition, if it is a malicious email, the case is exported to MISP.
7. If the verdict is not final, the analyst's intervention is required. He must review the case on TheHive along with the results given by the various analyzers to formulate a verdict, then he can send the notification to the user, optionally export the case to MISP and close the case.

ThePhish example usage

This example aims to demonstrate how a user can send an email to ThePhish for it to be analyzed and how an analyst can actually analyze that email using ThePhish.

A user sends an email to ThePhish

A user can send an email to the email address used by ThePhish to fetch the emails to analyze. The email has to be forwarded as an attachment in EML format so as to prevent the contamination of the email header. In this case, the used mail client is Mozilla Thunderbird and the used email address is a Gmail address.

<img src="pictures/demo/0_do_forward.png" width="400"> <img src="pictures/demo/1_forward.png" width="400">

The analyst analyzes the email

The analyst navigates to the web page of ThePhish and clicks on the "List emails" button to obtain the list of emails to analyze.

<img src="pictures/demo/2_gui_list.png" width="700">

When the analyst clicks on the "Analyze" button related to the selected email, the analysis is started and its progress is shown on the web interface.

<img src="pictures/demo/3_start_analysis_gui.png" width="700">

In the meantime, ThePhish extracts the observables (URLs, domains, IP addresses, email addresses, attachments and hashes of those attachments) from the email and then interacts with TheHive to create the case.

<img src="pictures/demo/4_created_case_ext.png" width="700">

Three tasks are created inside the case.

<img src="pictures/demo/5_tasks.png" width="700">

Then, ThePhish starts adding the extracted observables to the case.

<img src="pictures/demo/6_observables.png" width="700">

At this point the user is notified via email that the analysis has started thanks to the Mailer responder.

<img src="pictures/demo/7_mail_notification.png" width="400">

The description of the first task allows the Mailer responder to send the notification via email.

<img src="pictures/demo/8_task_notification.png" width="700">

After the first task is closed, the second task is started and the analyzers are started on the observables. The analysis progress is shown on the web interface while the analyzers are started.

<img src="pictures/demo/9_analyzing_gui.png" width="700">

The analysis progress can also be viewed on TheHive, thanks to its live stream.

<img src="pictures/demo/10_livestream.png" width="700">

Once all the analyzers have terminated their execution, the second task is closed and the third one is started, then ThePhish calculates the verdict. Since the verdict is "malicious", all the observables that are found to be malicious are marked as IoC. In this case only one observable is marked as IoC.

<img src="pictures/demo/11_ioc1.png" width="700">

The case is then exported to MISP as an event, with a single attribute represented by the observable mentioned above.

<img src="pictures/demo/12_created_misp_event.png" width="700"> <img src="pictures/demo/13_misp_attribute.png" width="700">

Then, ThePhish sends the verdict via email to the user thanks to the Mailer responder.

<img src="pictures/demo/14_mail_result.png" width="400">

Finally, both the task and the case are closed. The description of the third task allows the Mailer responder to send the verdict via email. Moreover, the case has been closed after five minutes and resolved as "True Positive" with "No Impact", which means that the attack has been detected before it could do any damage.

<img src="pictures/demo/15_16_task_case_closed.png" width="700">

Once the case is closed, the verdict is available for the analyst on the web interface together with the entire log of the analysis progress.

<img src="pictures/demo/17_malicious_verdict.png" width="700">

At this point the analyst can go back and analyze another email. The above-depicted case was related to a phishing email, but a similar workflow can be observed when the analyzed email is classified as "safe". Indeed, the case is closed and the verdict is sent via email to the user.

<img src="pictures/demo/18_mail_result_safe.png" width="400">

Then, the verdict is also displayed to the analyst on the web interface.

<img src="pictures/demo/19_safe_verdict.png" width="700">

On the other hand, when an email is classified as "suspicious", the verdict is only displayed to the analyst on the web interface.

<img src="pictures/demo/20_suspicious_verdict.png" width="700">

At this point the analyst needs to use the buttons on the left-hand side of the page to use TheHive, Cortex and MISP for further analysis. This is because the analysis has not been completed yet and so the user is only notified that the analysis of the email that he forwarded to ThePhish has been started. Indeed, the last task and the case have not been closed yet since they need to be closed by the analyst himself once he elaborates a final verdict.

<img src="pictures/demo/21_suspicious_task_open.png" width="700">

The analyst can view the reports of all the analyzers on TheHive and Cortex and, in case this revealed not to be enough, he could also download the EML file of the email and analyze it manually.

<img src="pictures/demo/22_eml_file.png" width="700">

When the analyst terminates the analysis, he can populate the body of the email to send to the user in the description of the last task, start the Mailer responder, export the case to MISP if the verdict is "malicious" by clicking on the "Export" button and then close the case.

Implementation

<img src="pictures/implementation.png" width="700">

ThePhish is a web application written in Python 3. The web server is implemented using Flask, while the front-end part of the application, which is the dynamic page written in HTML, CSS and JavaScript, is implemented using Bootstrap. Apart from the web server module, the back-end logic of the application is constituted by three Python modules that encapsulate the logic of the application itself and a Python class used to support the logging facility through the WebSocket protocol. If you want to see a graphi