Stock Analysis Engine

Build and tune investment algorithms for use with artificial intelligence (deep neural networks) <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/compose/docker/notebooks/Comparing-3-Deep-Neural-Networks-Trained-to-Predict-a-Stocks-Closing-Price-Using-The-Analysis-Engine.ipynb>__ with a distributed stack for running backtests using live pricing data on publicly traded companies with automated datafeeds from: IEX Cloud <https://iexcloud.io/>, Tradier <https://tradier.com/> and FinViz <https://finviz.com>__ (includes: pricing, options, news, dividends, daily, intraday, screeners, statistics, financials, earnings, and more).

Kubernetes users please refer to the Helm guide to get started <https://stock-analysis-engine.readthedocs.io/en/latest/deploy_on_kubernetes_using_helm.html>__ and Metalnetes for running multiple Analysis Engines at the same time on a bare-metal server <https://metalnetes.readthedocs.io/en/latest/#>__

.. image:: https://i.imgur.com/tw2wJ6t.png

Fetch the Latest Pricing Data

Supported fetch methods for getting pricing data:

• Command line using fetch command
• IEX Cloud Fetch API <https://stock-analysis-engine.readthedocs.io/en/latest/iex_api.html#iex-fetch-api-reference>__
• Tradier Fetch API <https://stock-analysis-engine.readthedocs.io/en/latest/tradier.html#tradier-fetch-api-reference>__
• Docker-compose using ./compose/start.sh -c
• Kubernetes jobs: Fetch Intraday <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/k8/datasets/pull_intraday_per_minute.yml>, Fetch Daily <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/k8/datasets/pull_daily.yml>, Fetch Weekly <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/k8/datasets/pull_weekly.yml>, or Fetch from only Tradier <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/k8/datasets/pull_tradier_per_minute.yml>

Fetch using the Command Line

Here is a video showing how to fetch the latest pricing data for a ticker using the command line:

.. image:: https://asciinema.org/a/220460.png :target: https://asciinema.org/a/220460?autoplay=1 :alt: Fetch Pricing Data using the Command Line

#. Clone to /opt/sa

::

    git clone https://github.com/AlgoTraders/stock-analysis-engine.git /opt/sa
    cd /opt/sa

#. Create Docker Mounts and Start Redis and Minio

This will pull `Redis <https://hub.docker.com/_/redis>`__ and `Minio <https://hub.docker.com/r/minio/minio>`__ docker images.

::

    ./compose/start.sh -a

#. Fetch All Pricing Data

#.  `Run through the Getting Started section <https://github.com/AlgoTraders/stock-analysis-engine#getting-started>`__

#.  Fetch pricing data from `IEX Cloud (requires an account and uses on-demand usage pricing) <https://iexcloud.io/cloud-login#/register/>`__ and `Tradier (requires an account) <https://developer.tradier.com/getting_started>`__:

    - Set the **IEX_TOKEN** environment variable to fetch from the IEX Cloud datafeeds:

    ::

        export IEX_TOKEN=YOUR_IEX_TOKEN

    - Set the **TD_TOKEN** environment variable to fetch from the Tradier datafeeds:

    ::

        export TD_TOKEN=YOUR_TRADIER_TOKEN

    - Fetch with:

    ::

        fetch -t SPY

    - Fetch only from **IEX** with **-g iex**:

    ::

        fetch -t SPY -g iex
        # and fetch from just Tradier with:
        # fetch -t SPY -g td

    - Fetch previous 30 calendar days of intraday minute pricing data from IEX Cloud

    ::

        backfill-minute-data.sh TICKER
        # backfill-minute-data.sh SPY

#.  Please refer to `the documentation for more examples on controlling your pricing request usage (including how to run fetches for intraday, daily and weekly use cases) <https://stock-analysis-engine.readthedocs.io/en/latest/scripts.html#module-analysis_engine.scripts.fetch_new_stock_datasets>`__

.. note:: Yahoo `disabled the YQL finance API so fetching pricing data from yahoo is disabled by default <https://developer.yahoo.com/yql/>`__

#. View the Compressed Pricing Data in Redis

::

    redis-cli keys "SPY_*"
    redis-cli get "<key like SPY_2019-01-08_minute>"

Run Backtests with the Algorithm Runner API

Run a backtest with the latest pricing data:

.. code-block:: python

import analysis_engine.algo_runner as algo_runner
import analysis_engine.plot_trading_history as plot
runner = algo_runner.AlgoRunner('SPY')
# run the algorithm with the latest 200 minutes:
df = runner.latest()
print(df[['minute', 'close']].tail(5))
plot.plot_trading_history(
    title=(
        f'SPY - ${df["close"].iloc[-1]} at: '
        f'{df["minute"].iloc[-1]}'),
    df=df)
# start a full backtest with:
# runner.start()

Check out the backtest_with_runner.py script <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/analysis_engine/scripts/backtest_with_runner.py>__ for a command line example of using the Algorithm Runner API <https://stock-analysis-engine.readthedocs.io/en/latest/algo_runner.html>__ to run and plot from an Algorithm backtest config file <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/cfg/default_algo.json>__.

Extract from Redis API

Once fetched, you can extract datasets from the redis cache with:

.. code-block:: python

import analysis_engine.extract as ae_extract
print(ae_extract.extract('SPY'))

Extract Latest Minute Pricing for Stocks and Options

.. code-block:: python

import analysis_engine.extract as ae_extract
print(ae_extract.extract(
    'SPY',
    datasets=['minute', 'tdcalls', 'tdputs']))

Extract Historical Data

Extract historical data with the date argument formatted YYYY-MM-DD:

.. code-block:: python

import analysis_engine.extract as ae_extract
print(ae_extract.extract(
    'AAPL',
    datasets=['minute', 'daily', 'financials', 'earnings', 'dividends'],
    date='2019-02-15'))

Additional Extraction APIs

• Extraction API Reference <https://stock-analysis-engine.readthedocs.io/en/latest/extract.html>__
• IEX Cloud Extraction API Reference <https://stock-analysis-engine.readthedocs.io/en/latest/iex_api.html#iex-extraction-api-reference>__
• Tradier Extraction API Reference <https://stock-analysis-engine.readthedocs.io/en/latest/tradier.html#tradier-extraction-api-reference>__
• Inspect Cached Datasets in Redis for Errors <https://stock-analysis-engine.readthedocs.io/en/latest/inspect_datasets.html#module-analysis_engine.scripts.inspect_datasets>__

Backups

Pricing data is automatically compressed in redis and there is an example Kubernetes job for backing up all stored pricing data to AWS S3 <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/k8/backups/backup-to-aws-job.yml>__.

Running the Full Stack Locally for Backtesting and Live Trading Analysis

While not required for backtesting, running the full stack is required for running algorithms during a live trading session. Here is a video on how to deploy the full stack locally using docker compose and the commands from the video.

.. image:: https://asciinema.org/a/220487.png :target: https://asciinema.org/a/220487?autoplay=1 :alt: Running the Full Stack Locally for Backtesting and Live Trading Analysis

#. Start Workers, Backtester, Pricing Data Collection, Jupyter, Redis and Minio

Now start the rest of the stack with the command below. This will pull the `~3.0 GB stock-analysis-engine docker image <https://hub.docker.com/r/jayjohnson/stock-analysis-engine>`__ and start the workers, backtester, dataset collection and `Jupyter image <https://hub.docker.com/r/jayjohnson/stock-analysis-jupyter>`__. It will start `Redis <https://hub.docker.com/_/redis>`__ and `Minio <https://hub.docker.com/r/minio/minio>`__ if they are not running already.

::

    ./compose/start.sh

.. tip:: Mac OS X users just a note that `there is a known docker compose issue with network_mode: "host" <https://github.com/docker/for-mac/issues/1031>`__ so you may have issues trying to connect to your services.

#. Check the Docker Containers

::

    docker ps -a

#. View for dataset collection logs

::

    logs-dataset-collection.sh

#. Wait for pricing engine logs to stop with ctrl+c

::

    logs-workers.sh

#. Verify Pricing Data is in Redis

::

    redis-cli keys "*"

#. Optional - Automating pricing data collection with the automation-dataset-collection.yml docker compose file <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/compose/automation-dataset-collection.yml>__:

.. note:: Depending on how fast you want to run intraday algorithms, you can use this docker compose job or the `Kubernetes job <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/k8/datasets/job.yml>`__ or the `Fetch from Only Tradier Kubernetes job <https://github.com/AlgoTraders/stock-analysis-engine/blob/master/k8/datasets/pull_tradier_per_minute.yml>`__ to collect the most recent pricing information

::

    ./compose/start.sh -c

Run a Custom Minute-by-Minute Intraday Algorithm Backtest and Plot the Trading History

With pricing data in redis, you can start running backtests a few ways:

• `Comparing 3 Deep Neural Networks Trained to Predict a Stocks Closing Price in a Jupyter Notebook <https://github.com/AlgoTraders/stock-analysis-engine/blob/