Stock Prediction using Neural Network on Amazon SageMaker

License Summary

This sample code is made available under a modified MIT license. See the LICENSE file.

Introduction

This is a sample workshop that demonstrates how to use a neural network-based algorithm for time series prediction. The workshop uses stock market data maintained by Deutsche Börse and made available through the Registry of Open Data on AWS. This dataset contains stock movement data from over 100 stocks traded on the Frankfurt Stock Exchange and is updated by the minute. Data is available starting from July 2016.

Time series data can be analyzed using a variety of techniques, including a simple multilayer perceptron (MLP), a stacked recurrent neural network (RNN), and forecasting methods such as Autoregressive Integrated Moving Average (ARIMA) or Exponential Smoothing (ETS). As a first attempt, we'll use a simple Recurrent Neural Network (RNN) model to predict the price of a single stock.

Action Plan

Amazon SageMaker is the Machine Learning platform on AWS that provides infrastructure to run hosted Jupyter Notebooks. Amazon SageMaker is integrated with other storage and analytics services on AWS to make the essential data management tasks for a successful Machine Learning project secure, scalable and streamlined.

In this workshop, we'll first use Amazon SageMaker-hosted notebooks to fetch the data from Deutsche Börse dataset, clean it, and aggregate it in Amazon S3 buckets.

In addition to hosted notebooks, Amazon SageMaker also provides managed training and hosting for machine learning models, using a variety of languages and libraries. Once we've prepared the data and stored it in Amazon S3, we'll use this functionality to containerize the machine learning training and prediction code, publish it on an Amazon Elastic Container Registry (Amazon ECR) repository, and host our custom model behind an Amazon SageMaker endpoint to generate predictions.

Amazon SageMaker also provides several built-in algorithms for image classification, regression, clustering of structured data, time series processing, and natural language processing. In the latter part of this workshop, we'll use DeepAR, which is a supervised learning algorithm for forecasting one-dimensional time series using RNN.

Disclaimer

This workshop is not an exercise in statistical methods, nor does it attempt to build a viable stock prediction model that you can use to make money. However, it does showcase the machine learning techniques that you can use on AWS.

1. Getting started

Since you will execute most of the workshop steps on a Jupyter Notebook hosted on Amazon SageMaker, start by creating a notebook instance on Amazon SageMaker from the AWS Console.

Refer to the AWS Region Table to check the availability of Amazon SageMaker, and choose to create the following infrastructure in any of the regions where it is available.

As of re:Invent-2018, Amazon SageMaker is available in the following regions:

• us-east-1 (Northern Virgina)
• us-east-2 (Ohio)
• us-west-1 (California)
• us-west-2 (Oregon)
• ca-central-1 (Canada)
• eu-west-1 (Ireland)
• eu-west-2 (London)
• eu-central-1 (Frankfurt)
• ap-northeast-1 (Tokyo)
• ap-northeast-2 (Seoul)
• ap-southeast-1 (Singapore)
• ap-southeast-2 (Sydney)
• ap-south-1 (Mumbai)
• us-gov-west-1 (AWS GovCloud)

1.1. Lifecycle configuration

Lifecycle configurations are small scripts, that you can use to automate certain tasks when a notebook instance is being created and/or being started. For this workshop, create a startup script to download pre-built notebooks from this Github repository onto your notebook instance.

Configure this script to run on Create notebook.

#!/bin/bash
set -e
git clone https://github.com/aws-samples/amazon-sagemaker-stock-prediction.git
mkdir SageMaker/fsv309-workshop
mv amazon-sagemaker-stock-prediction/container SageMaker/fsv309-workshop/container/
mv amazon-sagemaker-stock-prediction/notebooks SageMaker/fsv309-workshop/notebooks/
mv amazon-sagemaker-stock-prediction/images SageMaker/fsv309-workshop/images/
rm -rf amazon-sagemaker-stock-prediction
sudo chmod -R ugo+w SageMaker/fsv309-workshop/

<details> <summary><strong>Step-by-step instructions (expand for details)</strong></summary><p>

1. In the AWS Management Console choose Services then select Amazon SageMaker under Machine Learning.

2. Choose Lifecycle configurations under the section Notebook on the left panel.

3. Choose Create configuration to open the create dialog.

4. Type the name fsv309-lifecycle-config in the Name field.

5. In the tab Create notebook, type or copy-paste the Create Notebook script from above.

6. Finish configuration by clicking Create configuration.

</p></details>

<b>Note:</b> If you don't create a lifecycle configuration or attach the configuration to your notebook instance, you can always run the above commands directly into a Terminal window, from within your instance's Jupyter console.

1.2. Notebook instance

1. Use the lifecycle configuration to create a notebook instance in the region of your choice.

2. Choose a small instance class, such as ml.t2.medium. Since you won't use this notebook instance to execute training and prediction code, this will be sufficient.

3. If you do not have an AWS Identity and Access Management (IAM) role created prior with all the necessary permissions needed for Amazon SageMaker to operate, create a new role on the fly.

4. The IAM role you choose to use with the notebook needs to be authorized to create an Amazon ECR repository and upload an container image to the repository. Therefore add the following permissions to the IAM Role that you'll be using for your Notebook instance:

   • ecr:CreateRepository
   • ecr:InitiateLayerUpload
   • ecr:UploadLayerPart
   • ecr:CompleteLayerUpload
   • ecr:PutImage

5. Optionally you can choose to place your instance within a VPC and encrypt all data to be used within notebook to be encrypted. For the purpose of the workshop, you can proceed without doing this.

<details> <summary><strong>Step-by-step instructions (expand for details)</strong></summary><p>

1. In the AWS Management Console choose Services then select Amazon SageMaker under Machine Learning.

2. Choose Notebook instances under the section Notebook on the left panel.

3. Choose Create notebook instance to open the create dialog.

4. Type the name fsv309-notebook in the Name field.

5. From Notebook instance type dropdown, choose ml.t2.medium.

6. From IAM role dropdown, choose Create a new role.

7. In the dialog that pops up, select the radio button for Any S3 bucket, .

8. Choose Create Role to return to the notebook creation dialog. Notice that Amazon SageMaker creates a new execution role with the current timestamp appended at the end of its name, and that this role remains selected under IAM role dropdown.

9. From the Lifecycle configuration dropdown, choose the configuration named fsv309-lifecycle-config, that you created in section 1.1.

10. Immediately below the IAM Role field, you should see a success message in a green message box, with the name of your newly created IAM role displayed as a hyperlink. Click on the hyperlink to open the role in IAM console in a new browser tab.

11. From the IAM console page that opens in a new browser tab displaying the role summary, choose Add Inline policy

12. On the Create policy page, click on Choose a service

13. In the suggestive search box, type "EC2", to have the list of displayed service filtered down, then choose EC2 Container Registry from the narrowed down list.

14. Under the Actions section, expand Write Access level

15. Select actions - CreateRepository, InitiateLayerUpload, UploadLayerPart, CompleteLayerUpload and PutImage

16. Under the Resources section, click on the text displaying You chose actions that require the policy resource type.

17. Choose All resources radio button under Resources section.

18. Choose Review policy at the bottom right-hand corner of the screen.

19. On the review screen, ensure there are no errors or warnings displayed.

20. Type a name of the policy in Name field, Choose a meaningful name, such as ECRUpload.

21. Choose Create policy at the bottom right-hand corner of the screen.

22. Close the browser tab to return to the previous tab for the Amazon SageMaker console.

23. Leave the VPC selection and Encryption Keys empty for the purpose of this workshop, and choose Create notebook instance to finish creation.

24. You'll be returned to the list of notebooks, with the status of current notebook shown as Pending. Wait