TradingRLBot
A reinforcement learning agent trade eight different coins for several months using just an initial budget in cash.
Install / Use
/learn @Toroi01/TradingRLBotREADME
TradingRLBot
Table of contents
- Introduction and motivation
- Dataset
- Environment
- Models
- Evaluation
- Hyperparameter tuning
- Experiments
- How to run the code
- References
Introduction and motivation
2021 was a crazy year for Bitcoin and the cryptocurrency world in general. Bitcoin reached its all-time high of 64'000 USD and a lot of new exciting projects related to blockchain were developed. Several people start using these coins and trade them, attracted by the incredible rise in price. We decided then to test if the powerful neural networks could help us trading these coins.
The idea of the project is to make a reinforcement learning agent trade eight different coins for several months using just an initial budget in cash. As for the humans performing technical analysis, the agent has access just to some historical data of the price and volume.
Dataset
Data
The data that we used to build our dataset was downloaded from Binance, the biggest bitcoin exchange and altcoin crypto exchange in the world. The dataset provides the history of hourly prices in USD for the top 8 cryptocurrencies of the market ("BTC", "ETH", "BNB", "ADA", "XRP", "DOGE", "LINK", "LTC") and starts at 2020-01-01 00:00:00 and ends at 2021-06-30 23:00:00.
Preprocessing and feature engineering
In order to feed to the network clean and meaningful data we have performed the following operations:
- Fill missing values. We have filled the missing values uding the forward fill method, which propagates the last valid observation forward.
- Add technical indicators. These indicators are pattern-based signals produced by the variation in price and volume. We have divided these indicators into two types:
- Short term indicators To describe the variation of the price and value during the same day. We consider all the indicators offered by the library ta (over 90) and we selected the less correlated ones (cor <0.7) in the first month of data. We were left with 18 indicators.
- Long term indicators To describe what happened in the previous month we use four famous indicators: smooth moving average, directional moving index, relative strength index and commodity channel index. We chose three different windows: 1 day, 7 days and 30 days. In this way, we obtained other 12 technical indicators
- Add percentage change of the variables. To help the network to recognise the patterns, even if the absolute values are changing in time, we have calculated the daily change in percentage of price, volumes and all the technical indicators.
- Add covariance between the coins price. We have calculated the covariance among the different coins' closing prices, in order to provide information about the relationship among these cryptocurrencies.
- Eliminate the first 30 days of data. We couldnot consider the first 30 days since our metrics need a 30 days window period.
Environment
Our agents are trained in an environment developled by ourselves which emulates the trading options available in a crypto exchange.
Buy and Sell
The output of the agent is mapped to the environment actions that decide whether to Buy, Sell or Hold stock.
- Buy: If the agent has enough cash in the Portfolio, perform a buy. This means adding the bought assets to the portfolio while substracting their value and the comissions from the cash.
- Sell: If the agent has enough assets in the Portfolio, perform a sell. This means adding the value of the sold assets to the cash while substracting the amount of assets sold from the portfolio.
- Hold: Do nothing.
All actions are limited by a parameter called max_amount_per_trade, defined when creating the environment. This parameter helps to control that the algorithm does not trade with the whole portfolio value in each operation.
Action mapping
We need then to map the action performed by the agent to the environment action. This map is different depending if the output of the agent is discrete or continuous:
- Continuous actions: The action space is defined as a tensor of shape (1, number of assets), with values ranging from -1 to 1. The values are then multiplied by the max_amount_per_trade parameter. In example, havingthe actions of an environment with just three assets were represented as:
# With max_amount_per_trade = 1000
actions = [0.5, -0.1, 1]
# The actions would be mapped to:
# - Buy 50$ of the first asset
# - Sell 10$ of the second asset
# - Buy 100$ of the third asset
- Discrete actions: The output of the models is defined as a integer ranging from [0, 2*number of assets]. A value of 0 means holding. Odd numbers are buy operations and even numbers are sell ones. In this case, each operation trades with the whole max_amount_per_trade.
Updating the Portfolio
The Portfolio object holds the trading logic and keeps track of how the capital is allocated across the different assets at each point. At the beggining of an experiment everything is in cash, but with every buy and sell the allocations are modified.
At the end of the experiment, the capital that the Portfolio is holding can be compared to the one in the initial stat to measure the return of the startegy. It can also show when and which the transactions were done.
Updating the state
The state in each timestep is defined bya vector composed of:
- The amount of cash in the portfolio
- The amount of the assets in the portfolio
- The price of assets (close values)
- The features built for all assets in the preprocessing step
In every new timestep we need to update all this values since the market conditions and the portfolio allocations might have changed.
Computing the reward
Agents are rewarded at every timestep with the difference between the value of the portfolio before and after performing the actions. At the beginning we were using the difference in absolute value (portfolio_after - portfolio_before), but then we decided to use the change in percentage ((portfolio_after - portfolio_before)/portfolio_before )). With the second approach we obtained as expected better results, since the return in percentage is independent of the size of the portfolio which can change a lot in different time period.
Other reward functions have been implemented (Sharpe, sortino) but there was no time to test them. These functions would have been more aligned with our trading strategy. More information in the Metrics section.
Models
We have tried three different models, one uses the discete action space previously defined (DQN) and the other two the continuous action space (DDPG and PPO).
DQN
The deep Q-Network is a off-policy Q-learning which use different tricks to stabilize the learning like a replay buffer and a target network. To have a nice introduction to Reinforment learning and DQN you can look at the following links: a-long-peek-into-reinforcement-learning. For the network we have left the default architecture proposed by the paper Human-level control through deep reinforcement learning: two hidden layers with 64 neurons each, RELUs for all hidden layers and a tanh activation in the last layer. The optimizer used is ADAM.
DDPG
The deep deterministic policy gradient is a model free, off-policy algorithm which combines DPG and DQN. To know more about the policy gradient algorithms you can look at this blog: policy-gradient-algorithms. For all the networks involved we have left the default architecture proposed by the paper Continuous control with deep reinforcement learning: two hidden layers of respecively 400 and 300 neurons, ReLUs for all hidden layers and the final output layer of the actor was a tanh layer, to bound the actions. The optimizer used is ADAM
